High-order modulation formats, constellation design, and digital signal processing for high-speed transmission systems

Abstract The capacity of optical fiber communication networks is limited by the Kerr effect inherent to transmission using optical fibers. The signal degradations due to the nonlinear distortions limit the achievable transmission distances and become more significant in systems with larger transmission bandwidths, closer channel spacing, and higher order modulation formats. Optical fiber nonlinearities are seen as the major bottleneck to the performance of optical transmission networks. This chapter describes the theoretical and experimental investigations into a series of techniques developed to unlock the capacity of optical communications and to overcome the barriers in transmission over nonlinear fiber channels. It covers three key areas for combatting optical fiber nonlinearities to increasing the overall throughput of the optical fiber channels in the nonlinear regime that have been the focus of research over the recent years. These are (1) digital nonlinearity compensation technique, such as digital backpropagation, to partially “undo” the nonlinearity and improve signal performance, (2) digital nonlinearity compensation in presence of laser phase noise, and (3) signal design techniques, making use of coded modulation and constellation shaping in optical communications. This chapter aims to review and quantify examples of digital signal processing-based nonlinearity compensation and further possible increases in the achievable capacity and transmission distances, depending on the modulation format used, that come from the combination of nonlinearity compensation and signal constellation shaping.