Providing In-network Support to Coflow Scheduling

Emerging distributed applications, such as big data analytics, generate a large number of flows that concurrently transport data across data center networks. To improve their performance, it is required to account for the behavior of such a collection of flows, i.e., coflows, rather than individual ones. State-of-the-art solutions achieve near-optimal completion time by continuously reordering unfinished coflows at the end-host and using network priorities.This paper shows that dynamically changing flow priorities at the end-host, without considering in-flight packets, can cause high degrees of packet reordering, thus imposing pressure on the congestion control and potentially harming network performance in the presence of switches with shallow buffers. We present pCoflow, a new solution that integrates end-host based coflow ordering with in-network scheduling based on packet history. Our evaluation shows that pCoflow improves in coflow completion time upon state-of-the-art solutions by up to 34% for varying loads.