A number of efforts have been made to measure the performance of ORBs, often comparing with performance of other ORBs [23]. Earlier studies on the performance of CORBA objects focussed mainly on identifying the performance constraints of an Object Request Broker (ORB) alone. Schmidt analyzed the performance of Orbix and VisiBroker over high speed ATM networks and pointed out key sources of overhead in middleware ORBs [7,8]. This paper complements Schmidt's work by demonstrating an integrated and automated approach which is capable of simultaneously measuring the influence of the ORB, the operating system and the underlying network on the performance of CORBA objects.
Studies have also been conducted on IDL compiler optimizations that can improve overall performance of the ORB. One such effort is the Flick project[4]. It is claimed that Flick-generated stubs marshal data between 2 and 17 times faster than stubs produced by traditional IDL compilers, resulting in an increased end-to-end throughput by factors between 1.2 and 3.7. While clearly addressing an important topic, the Flick work also clearly concentrates on one specific facet of endsystem performance. Our work complements such efforts by providing a platform within which the effect of such efforts on endsystem performance can be evaluated.
TAO is the ACE ORB being developed at Washington University [17]. This project focuses on: (1) identifying the enhancements required to standard ORB specifications that will enable applications to specify their Quality of Service (QoS) requirements to ORBs, (2) determining features required to build real-time ORBs, (3) integrating the strategies for I/O subsystem architectures and optimizations with ORB middleware, and (4) to capture and document key design patterns necessary to develop, maintain and extend real-time ORB middleware. The work described in this paper compliments these goals by providing a way to capture, document, and evaluate performance aspects of ORB based design patterns.
In addition to providing a real-time ORB, TAO is an integrated ORB endsystem architecture that consists of a high-performance I/O subsystem and an ATM port inter-connect controller (APIC). They have developed a wide range of performance tests which include throughput tests[7], latency tests[8] and demultiplexing tests[8]. They have used these performance tests to test TAO [17] and other CORBA2.0 compliant ORBs. Their tests formed a basis for several of the basic tests in the automated framework described in this paper.
A commercially available CORBA test suite is the VSORB from X/Open [28]. VSORB is implemented under the TETware test harness, a version of the Test Environment Toolkit (TET), a widely used framework for implementing test suites [24]. It is designed for two primary uses: (1) testing ORB implementations for CORBA conformance and interoperability under formal processes and procedures, and (2) CORBA compliance testing by ORB implementors during product development and quality assurance. This work differs from ours in that it concentrates on compliance rather than performance, but clearly shares the goal of creating a general framework for large scale evaluation tests.
The Manufacturing Engineering Laboratory at the National Institute of Standards & Technology(NIST) takes a different approach towards the benchmarking of CORBA in their current work on the Manufacturer's CORBA Interface Testing Toolkit(MCITT) [13]. They use a emulator-based approach in which the actual servers are replaced by test servers and the person doing the testing only needs to specify the behaviors that are important for the specific scenario being examined. The approach provides an extremely simplified procedural language, the Interface Testing Language, for specifying and testing the behavior of CORBA clients and servers. This work is similar to ours with respect to its abstraction of the object behavior, but it does not explicitly integrate endsystem evaluation, concentrating only on the application and ORB middleware.
The Distributed Systems Research Group at Charles University, Czech Republic, have done a comparison of three ORBs based on a set of criteria including dispatching ability of the ORB, throughput provided for the invocation of different data types, scalability, and performance implications of different threading architectures [2,19]. The criteria address different aspects of the ORB functionality and the influence of each criterion has been discussed with respect to specific ORB usage scenarios. They have also developed a suite of benchmarking applications for measurement and analysis of ORB middleware performance. This is a strong effort, but the drawback to this approach, in our view, is that it is restricted to evaluating ORB level performance and specific predefined application scenarios. This is significant because application behaviors will vary and their method does not appear, by our understanding, to be designed to support user specified test scenarios.
Performance evaluation is an important topic in many areas of computer system design and implementation, and significant related work exists which does not consider ORB performance. Data bases provide some of the best developed examples of benchmarks addressing application scenarios. It is interesting to observe that both data bases and ORBs support applications by assuming the role of middleware. As such, performance evaluation of data bases is most meaningful and useful to potential users when it considers application scenarios.
The Wisconsin Benchmark is an early effort to systematically measure and compare the performance of relational database systems with database machines[3]. The benchmark is a single-user and single-factor experiment using a synthetic database and a controlled workload. It measures the query optimization performance of database systems with 32 query types to exercise the components of the proposed systems. This is similar to our effort in that it abstracts the application scenario and considers a range of system functions. Our work differs, however, in that we also provide for placing the set of ORB based objects in an endsystem context including background load and traffic.
The ANSI SQL Standard Scalable and Portable Benchmark (AS3AP) models complex and mixed workloads, including single-user and multi-user tests, as well as operational and functional tests [25]. There are 39 single-user queries consisting of utilities, selection, join, projection, aggregate, integrity, and bulk updates. The four multi-user modules include a concurrent random read test and a pure information retrieval (IR) test[26]. The concurrent random write test is used to evaluate the number of concurrent users the system can handle updating the same relation. The mixed IR test and the mixed OLTP test are to measure the effects of the cross-section queries on the system with concurrent random reads or concurrent random writes. This effort has a stronger similarity to ours in that it considers a wider range of activity as well as multiple users. It does not, to our knowledge, provide support for users to specify application based test scenarios.