Direction-of-Arrival Estimation in a Mixture of Multiple Circular and Non-Circular Signals Using Nested Array

In practice, there are many circular and non-circular signals due to multipath propagation and various modulations. Conventional direction-of-arrival (DOA) estimation in a mixture of circular and non-circular signals cannot distinguish two kind of signals and detect more sources than number of sensors at the same time. This paper proposes a novel separation algorithm based on elliptic covariance matrix (ECM) which possesses accurate DOA estimation and high degrees of freedom (DOF) with low complexity. Firstly it estimates non-circular signals using ECM which contains non-circular information merely. Considering that the virtual array generated from nested array using ECM is inconsecutive, a matrix completion method via nuclear norm minimization is also included and as a result, the freedom degrees are further extended. On the basis of ECM, the paper also introduces a separation algorithm through subtraction of two reconstructed Toeplitz covariance matrix (CM). Detailed analysis and theoretically proof is presented subsequently and DOAs of circular signals can be obtained after separation. Simulation results show that the proposed algorithm can realize underdetermined estimation and get accurate DOAs while two kind of signals are separated simultaneously.