We propose the use of a new protocol which operates between the network layer (IP) and transport layer (TCP, UDP, etc.) that addresses the need for transport-level coordination. We call this protocol the Coordination Protocol (CP). The coordination function provided by CP is transport protocol independent. At the same time, CP is distinct from network-layer protocols like IP that play a more fundamental role in routing a packet to its destination.
CP works by attaching probe information to packets transmitted from one cluster to another. As additional probe information is returned along the reverse cluster-to-cluster data path, a picture of current network conditions is formed by the AP and shared among endpoints within the local cluster. A consistent view of network conditions across flows follows from the fact that the same information is shared among all endpoints.
Figure 3 shows our proposed network architecture from a stack implementation point of view. CP exists on each endpoint device participating in the C-to-C application, as well as on the two aggregation points (APs) on either end of the cluster-to-cluster data path. Routers on the data path between APs need not be CP-enabled since they examine only the IP header of each incoming packet in order to route the packet in their customary manner.
The decision to insert CP between the network and transport layer rather than handling coordination at the application level requires some justification. Of primary importance to us is the preservation of end-to-end semantics. An alternative would be for each endpoint to send to a multiplexing agent who would send the data, along with probe information, to a demultiplexing agent on the remote cluster. By breaking the communication path into three stages, however, the end-to-end semantics of individual transport-level protocols have been severed. Such a scheme would also mandate that application-level control is centralized and integrated into the multiplexing agent.
Furthermore, we note that CP logically belongs between the network and transport layer. While the network layer handles the next-hop forwarding of individual packets and the transport layer handles the end-to-end semantics of individual streams, CP is concerned with streams that share a significant number of hops along the forwarding path but do not share the same end-to-end path. This relaxed notion of a stream bundle logically falls between the strict end-to-end notion of the transport-level and the independent packet notion of the network-level.
Finally, placement of CP between the network and transport layer allows for greater efficiency. In an application-level implementation of CP, information on network conditions (e.g., round trip time between APs) must pass up through an endpoint's protocol stack to the application layer. The information must then be passed back down to the transport layer where sending rate adjustments can be made in response to the information. In contrast, a distinct coordination layer allows for the information to be received and passed directly to the transport layer in a single pass as the incoming packet is processed by each layer of its endpoint's network stack.
While we acknowledge that implementing CP mechanisms at the application layer is indeed possible, we believe there are distinct advantages to the approach we have chosen. We emphasize, however, that the relative merits or drawbacks of our scheme are merely implementation issues that should not obscure the fundamental problem of C-to-C flow coordination described in this paper.
