Yuan Sun Irfan Sheriff Elizabeth M. Belding-Royer Kevin C. Almeroth
Department of Computer Science
University of California, Santa Barbara
{suny, isheriff, ebelding, almeroth}@cs.ucsb.edu
The majority of wireless research to date has been conducted using simulations which offer an efficient and flexible means to evaluate new protocols using fine-grained control. However, in simulations, MAC protocol models are often simplified, ideal wireless channels are assumed without consideration of background noise and random interference, and unrealistic traffic traces are utilized. Consequently, evaluation through simulation may not reflect the performance obtained in real networks.
As a result of the inaccuracy of simulations, many researchers have begun deploying multi-hop mesh networks for use in wireless network protocol development and testing. Testbed experiments can be challenging due to the effort required to install, configure and manage the hardware [5]. In addition, performance results are often affected by the specific configurations and protocol settings. Given the significant number of possible parameters that can affect results, finding a representative set of parameter values is non-trivial. Furthermore, the highly varying characteristics of wireless links often lead to unstable and unrepeatable results. Significant effort is necessary to enable repeatable tests and to establish adequate methods for collecting and analyzing the testbed data.
In this paper, we present our experimental study of multimedia traffic performance in mesh networks. Multimedia applications are examined because they represent a growing percentage of Internet traffic and applications. These applications demand more stringent service quality with low delay and jitter. Specifically, we perform tests consisting of video streams and voice traffic over the UCSB MeshNet testbed (https://moment.cs.ucsb.edu/meshnet). We evaluate the performance of the delivery of the multimedia data through multi-hop wireless paths and study the capacity of the mesh network. We also examine the impact of different traffic and network characteristics on application performance. Specifically, we compare the performance of bursty video traffic with constant bit rate voice traffic. We also investigate the impact of different wireless network interface card configurations. We believe our study is beneficial in both wireless network capacity planning and protocol design. We describe our analysis methodology and utilities so that other researchers can draw upon our experience for their own mesh testbed experiments.
The remainder of this paper is organized as follows. Section 2 briefly introduces the UCSB MeshNet testbed and describes the set of tools we used for our experiments. Section 3 describes the experimental setup and the evaluation metrics. Section 4 presents the experimental results and performance analysis. Finally, Section 5 discusses our observation and concludes the paper.
The UCSB MeshNet testbed is a wireless mesh network deployed on the campus of UC Santa Barbara. The network consists of 25 nodes equipped with IEEE 802.11b wireless radios. The nodes are distributed on five floors of the Engineering I building. The purpose of the testbed is to evaluate protocols and systems designed for the robust operation of multi-hop wireless networks.
The UCSB MeshNet testbed consists of two different types of nodes. Our experiments are conducted on one type of node, called Mesh Gateways, which are off-the-shelf Intel Celeron 2.4GHz machines running Linux version 2.4.20. The machines use wireless utilities version 16 and the hostap driver for communicating with the Netgate 2511 PCMCIA 802.11b radios. The 802.11b radios operate in ad hoc mode and connect the wireless mesh nodes. Each node is also equipped with an Ethernet interface to provide Internet access to the mesh devices and to allow out-of-band management of the mesh gateway [5].
We utilize existing tools such as iwpriv to set the pseudo BSSID and lock the cell because otherwise BSSID changes occur frequently in the tests and significantly impact the results. iptables is also used for packet filtering and route configurations. To facilitate repeatable experiments and accurate data analysis, we also developed two utilities for network monitoring and diagnosis.
Link reliability test tool:
We perform link reliability tests between node pairs. The
goals are to 1) measure the link quality of individual hops, and 2)
identify any asymmetric links. To test reliability,
the packet delivery rate in both the forward and backward direction of a link
are measured. The measurements are done
by sending periodic broadcast packets and recording
the number of packets successfully received at each
neighbor during a given period of time.
Broadcast packets are used because MAC layer retransmissions do not occur
for broadcast packets and thus these packets
can be used to estimate the raw packet delivery rate.
A link is considered symmetric if
the packet delivery rate on both the forward and reverse path
is above 70%.
We perform each test multiple times and identify node pairs that have reliable bi-directional links.
We use these node pairs for our experiments.
We also verify the reliability of
the links before and after each test run to ensure that the link quality is consistent with
our long term measurements.
Time synchronization tool:
In our performance analysis, time synchronization between the mesh nodes is needed
for delay and bandwidth calculations. The multimedia traffic cannot be utilized itself because it uses UDP as the
transport layer protocol. It is thus
one-way, i.e., no ACKs are provided. Therefore, the packet transmission delay
needs to be measured by the destination.
Further, because asymmetric links
frequently occur in wireless networks, round trip latency does not provide a consistent,
accurate measurement of one-way delay.
Therefore, time synchronization is critical for mesh testbed
experiments.
We initially applied the Network Time Protocol (NTP) [4] to eliminate the clock skew among the mesh nodes. However, our results show that the NTP synchronization precision is tens of milliseconds. This level of accuracy is not sufficient for our data analysis. Thus, we developed a tool to calculate the time difference of two machines by utilizing the wired management links of the mesh nodes. These links connect to the local area network in the Engineering I building. Specifically, our tool transmits consecutive 4-byte probe packets that include the timestamp of the source node. Upon reception of these probing packets, the destination node records the timestamp and echos a 4-byte packet containing the time difference between the two timestamps. At the same time, the source also sends 4-byte probe packets to measure the round trip latency. The real clock difference between the two nodes is the difference transmitted by the destination minus half of the round trip latency. We repeat the tests ten times. Our measurements indicate that, in the local area wired network, the average round trip latency and the time difference calculation have less than 10 error.
We examine the performance of multimedia traffic over the mesh network. Specifically, we use UDP video and voice streams recorded with RTPtools [1]. We use rtpplay for streaming at the source node and rtpdump to record the packets received at the destination. Voice traffic follows a constant bit rate (CBR) with an 80-byte voice packet transmitted every using G.711 codec, resulting in a data rate of . Video traffic, on the other hand, tends to be more bursty. Figure 1 plots a 10-second sample trace of a video source. The source transmits between two and three frames of data every second, where each frame consists of between three and seven packets. These packets are typically sent within a couple of milliseconds. H.261 codec is used for the video traffic and the average bit rate is .
We conduct the following set of experiments:
The metrics used to evaluate performance are:
In this section, we evaluate the performance of video and audio traffic through multi-hop wireless paths and study the capacity of the mesh network. We also examine the impact of different traffic and network characteristics on the application performance. Further, we show the impact of different wireless network interface card configurations.
Table 1 shows the number of video and voice flows that the mesh
network supports as the number of hops increases.
For video data, we consider less than packet loss acceptable. If a more
resilient coding scheme is utilized, it is possible that a higher loss rate will be tolerable.
For voice data, we consider as the interactive
voice delay threshold [3].
We tested the performance with the
the NIC set to a fixed data rate () and with auto-rate adaptation.
Intuitively, the network should support more voice traffic flows than video traffic because voice uses a lower date rate. However, as can be seen in Table 1, this is not the case. Instead, the packet sennding rate plays a more important role in determining the capacity. Specifically, voice traffic has a higher packet generation rate of 100 pkts/second, while the bursty video traffic has an average rate of about 16 pkts/second. The higher sending rate leads to network congestion, while the packet size has negligible impact on the number of supportable flows in the network. To verify the impact of packet sizes, we also performed experiments with byte voice packets. This results in a bit rate of . The results indicate that the same number of flows are supported regardless of whether the rate is or .
Figure 2 shows the average packet delivery latency and loss rate for video traffic with a fixed data rate (Figures 2(a) and (d)) and auto-rate (Figures 2(b) and (e)). We do not include the results for voice traffic because they are similar except that voice traffic in general incurs low delivery latency due to small packet sizes.
We observe that as the length of the transmission path increases, the performance degrades and the latency and loss rate increase. However, the increase is non-linear due to the increased interference from neighboring nodes. The network capacity is constrained by the number of hops. From the results, we also observe that increasing the transmission rate of the card does not necessarily increase the capacity. For instance, the number of flows supported with the auto-rate feature (with maximum rate = ) is close to that of the fixed data rate () in multi-hop scenarios, especially for voice flows with a large hop count. This result occurs because of the increased contention from neighboring nodes when the path length increases. With more packet contention and subsequent packet loss, the card will automatically fall back to a lower transmission rate.
In our experiments, we notice that the auto-rate adaptation follows a slow-start-like process. All nodes operate at the lowest data rate initially. We also occasionally observe a surprisingly low video flow delivery rate for a small number of flows in the auto-rate scenario. This is because auto-rate does not always succeed in adapting to a higher transmission rate when traffic is bursty. However, once the card succeeds in adaptation, a close-to-optimal throughput of about can be achieved [2].
Figures 2(c) and (f) compares the performance obtained during the day and at night for video flows traversing two hops with auto-rate. Interestingly, although our test nodes operate on a different channel than the other wireless networks in the building, we notice random interference and background noise during the day that significantly impacts the results.
We believe our study is beneficial for both wireless network capacity planning and protocol design. We have described our analysis methodology and utilities so that other researchers can draw upon our experience for their own mesh testbed experiments. We plan to continue our work studying experimental results obtained through our testbed. Specifically, we want to explore the techniques of reducing packet jitter in multimedia delivery and apply more advanced codec schemes and subjective evaluation methods to our traffic analysis.