Optimizing Deadline-Driven Bulk Data Transfers in Overlay Networks

Deadline-driven bulk data transfers frequently occur in overlay networks running data-intensive, dis- tributed workflow applications, such as grid and cloud envi- ronments. What distinguishes such transfers from other In- ternet traffic is that overlay nodes should cooperate towards the common goal of delivering all inter-dependent data timely, rather than follow individual, selfish goals. For such scenarios, we propose scheduling transfers in overlays in a globally optimal manner with respect to minimizing overall network congestion. Our optimization jointly addresses routing of transfers within the overlay and the time-domain scheduling of transfer bandwidths. We formally define and address the associated problem, the Bulk Data Routing and Transfer(BDRT) and present a linear programming-based solution to it, optimal in both routing and time domains. We additionally explore alternative approaches based on heuristic routing strategies, both oblivious and time-domain optimized. We evaluate these solutions via both PlanetLab trace-driven simulations and Internet transfer experiments, on the Intrigger wide-area grid and PlanetLab. Evaluation shows that our approach finds optimal solutions, based on estimations of job arrival times, deadlines and transfer volumes. Index Terms—congestion minimization, deadline-driven transfers, routing overlays, Internet I. INTRODUCTION