DOA estimation of coherently distributed sources in massive MIMO systems with unknown mutual coupling

Abstract In this paper, a direction of arrival (DOA) estimation algorithm for coherent distributed (CD) sources considering unknown mutual coupling in massive MIMO systems is proposed. By utilizing an enhanced sample covariance matrix (SCM) that matches well with massive MIMO systems, the proposed algorithm constructs a perturbed model and further exploits the sparse total east squares (STLS) technique to achieve DOA and angular spread estimation. With the DOA and angular spread estimates, the mutual coupling coefficients are given by solving the constrained minimization problem. Compared with the other state-of-the-art algorithms, the proposed algorithm not only can provide improved DOA estimation performance, but also can conquer the deficient scenario in a much easier way. Theoretical performance analysis and numerical examples validate the effectiveness of the proposed algorithm.