Bayesian factor analysis and performance measurement of the Linux forwarding architecture

Linux software routers have many configuration options, most of which have received little attention from researchers. Over time, multiqueue NICs, NUMA architectures, and many changes to the kernel have altered the forwarding landscape. Here we investigate (i) allocation of NIC queues to processing cores, (ii) batch sizes at various layers, (iii) receive and transmit packet steering, and (iv) how FIB performance scales. Our experiments focus on forwarding minimum size packets, which maximize the packet processing stress for the kernel, at 10 Gbps. Our investigation uses Bayesian factor analysis, experimental design techniques, and kernel density estimation to robustly measure each factor's effect on the mean packet forwarding rate and mean RTT. This study does not seek bleeding edge performance. Rather, we elucidate key decisions for packet forwarding performance on a widely used platform. Our results show irqbalance is quite volatile, hyperthreading can provide over a 1 MPPS boost in forwarding performance, larger queue sizes do not improve forwarding but can add non-trivial latency (up to 30 ms), and both receive packet steering and low values of rx-usecs can induce receive livelock.