Nonlinear pre-coding apparatus of multi-antenna system, has pre-coding unit that extents original constellation points of modulated symbols to several constellation points by using limited perturbation vector