The competition between the intrinsic and Rashba spin-orbit coupling and effects of correlations on Rashba SOC-driven transitions in the Kane-Mele model

We investigate, firstly, the effects of the Rashba SOC on the band structrue of the Kane-Mele model. The competition between the Rashba SOC and the intrinsic SOC can lead to the rich phenomenology. The Rashba SOC can drive the indirect and direct energy gap to close successively, but maintain the band touching between the valence band and the conduction band when the Rashba SOC is large enough to dominant the competition. We find that these touching points are located at $K$ and $K^{\prime}$ or/and some $2\pi/3$ rotationally symmetric points around $K$ and $K^{\prime}$ in the Brillouin zone. The indirect and direct energy gap closings correspond to the topologically trivial and non-trivial phase transitions respectively. For the small intrinsic SOC, the topologically non-trivial transition occurs when the ratio of the Rashba SOC to the intrinsic SOC is equal to the classical result, i.e. $2\sqrt{3}$. For the large intrinsic SOC, however, we find that the ratio decreases with the increasing intrinsic SOC. Secondly, using the slave-rotor mean field method we investigate the influences of the correlation on the Rashba SOC-driven topologically trivial and non-trivial transition in both the charge condensate and Mott regions. The topological Mott insulator with gapped or gapless spin excitations can arise from the interplay of the Rashba SOC and correlations.