Application of the carleman boundary-value problem to the investigation of wave propagation in a medium with smooth transition