High Performance Commodity Networking in a 512-CPU Teraflop Beowulf Cluster for Computational Astrophysics

We describe a new 512-CPU Beowulf cluster with Teraflop performance dedicated to problems in computational astrophysics. The cluster incorporates a cubic network topology based on inexpensive commodity 24-port gigabit switches and point to point connections through the second gigabit port on each Linux server. This configuration has network performance competitive with more expensive cluster configurations and is scaleable to much larger systems using other network topologies. Networking represents only about 9% of our total system cost of USD$561K. The standard Top 500 HPL Linpack benchmark rating is 1.202 Teraflops on 512 CPUs so computing costs by this measure are $0.47/Megaflop. We also describe 4 different astrophysical applications using complex parallel algorithms for studying large-scale structure formation, galaxy dynamics, magnetohydrodynamic flows onto blackholes and planet formation currently running on the cluster and achieving high parallel performance. The MHD code achieved a sustained speed of 2.2 teraflops in single precision or 44% of the theoretical peak. The Olympic motto ”Citius, altius, fortius” (”Swifter, higher, stronger”) succinctly describes the direction of 21st century parallel supercomputing swifter processors, higher resolution and stronger fault-tolerant parallel algorithms. In computational astrophysics, the need for all of these qualities is perhaps greater than most supercomputing applications. While the bulk of the physics of the formation of the planets, stars, galaxies and the large-scale structure in the universe are now largely understood with the initial conditions well posed in some cases, the challenge of computing the formation of objects and structures in the universe is difficult. The standard methods of computational fluid dynamics (including magnetohydrodynamics) and gravitational N-body simulation are stressed by the large dynamic range in density, pressure, and temperature that exist in nature. Everfiner computational meshes and greater numbers particles in N-body simulations are needed to capture the physics of the formation of things in the universe correctly.