Interlayer RKKY coupling in bulk Rashba semiconductors under topological phase transition

The bulk Rashba semiconductors BiTeX (X=I, Cl and Br) with intrinsically enhanced Rashba spin-orbit coupling provide a new platform for investigation of spintronic and magnetic phenomena in materials. We theoretically investigate the interlayer exchange interaction between two ferromagnets deposited on opposite surfaces of a bulk Rashba semiconductor BiTeI in its trivial and topological insulator phases. In the trivial phase BiTeI, we find that for ferromagnets with a magnetization orthogonal to the interface, the exchange coupling is reminiscent of that of a conventional three-dimensional metal. Remarkably, ferromagnets with a magnetization parallel to the interface display a magnetic exchange qualitatively different from that of conventional three-dimensional metal due to the spin-orbit coupling. In this case, the interlayer exchange interaction acquires two periods of oscillations and decays as the inverse of the thickness of the BiTeI layer. For topological BiTeI, the magnetic exchange interaction becomes mediated only by the helical surface states and acts between the one-dimensional spin chains at the edges of the sample. The surface state-mediated interlayer exchange interaction allows for the coupling of ferromagnets with non-collinear magnetization and displays a decay power different from that of trivial BiTeI, allowing the detection of the topological phase transition in this material. Our work provides insights into the magnetic properties of these newly discovered materials and their possible functionalization.