Spin liquid versus long range magnetic order in the frustrated body-centered tetragonal lattice

We show how spin-liquid (SL) states can be stabilized in a realistic three-dimensional model as a result of frustration. $\text{SU}(n)$-symmetric generalization of the Heisenberg model for quantum spin $S$ operators is used to investigate the frustrated body-centered tetragonal (BCT) lattice with antiferromagnetic interlayer coupling ${J}_{1}$ and intralayer first and second-neighbor couplings ${J}_{2}$ and ${J}_{3}$. By using complementary representations of the spin operators, we study the phase diagram characterizing the ground state of this system. For small $n$, we find that the most stable solutions correspond to four different families of long-range magnetic orders that are governed by ${J}_{1},\phantom{\rule{0.16em}{0ex}}{J}_{2}$, and ${J}_{3}$. First, some possible instabilities of these phases are identified for $n=2$, in large $S$ expansions, up to the linear spin-wave corrections. Then, using a fermionic representation of the $\text{SU}(n)$ spin operators for $S=1/2$, we find that purely magnetic orders occur for $n\ensuremath{\le}3$ while SL solutions are stabilized for $n\ensuremath{\ge}10$. The SL solution governed by ${J}_{1}$ breaks the lattice translation symmetry. The modulated SL is associated with a commensurate ordering wave vector $(1,1,1)$. For $4\ensuremath{\le}n\ensuremath{\le}9$, we show how the competition between ${J}_{1},\phantom{\rule{0.16em}{0ex}}{J}_{2}$, and ${J}_{3}$ can turn the magnetically ordered ground state into a SL state. Finally, we discuss the relevance of this scenario for correlated systems with BCT crystal structure.