Topological Mott insulator phase and metal-insulator crossover in organic Dirac electron system $\alpha$-(BEDT-TSeF)$_2$I$_3$

To elucidate the low-temperature ($T$) insulating state of $\alpha$-(BEDT-TSeF)$_2$I$_3$ ($\alpha$-(BETS)$_2$I$_3$), we construct a two-dimensional extended Hubbard model with transfer integrals including spin-orbit-coupling (SOC) and Coulomb interaction, based on first-principles calculation. We investigate the low-$T$ electronic state using Hartree Fock approximation, and found that the topological Mott insulator (TMI) state is a candidate for the insulating state. In this state, energy gap in the high-$T$ weak topological insulator (TI) state is strongly enhanced by next-nearest-neighbor interactions $V'$. With the increase in $T$, the TMI state continuously changes to a high-$T$ weak TI state. Calculations using random-phase-approximation also indicate that the ferrimagnetic spin fluctuation appears in the high-$T$ TI state, when weak onsite interaction is considered.