Low complexity frequency-response masking filters using modified structure based on serial masking

This paper presents a technique to synthesize low-complexity high-selectivity arbitrary-bandwidth finite impulse response (FIR) digital filters using the frequency-response masking (FRM) technique. A serial masking scheme is used to perform the masking task in two stages lowering the complexity of the masking filters. The bandedge shaping filter is implemented as a prefilter-equalizer cascade lowering the complexity of the band-edge shaping filter. The design of the FRM filter is formulated as a nonlinear nonconvex semi-infinite programming optimization problem whereby the maximum frequency response magnitude error of the overall filter is minimized. A constraint transcription method and a smoothing technique are employed to transform the continuous inequality constraints into equality constraints. By using the concept of penalty function, the transformed constraints are incorporated into the cost function to form a new cost function. The nonlinear optimization problem with inequality constraints is approximated by a sequence of unconstraint optimization problems which can be solved efficiently by a limited-memory BFGS (L-BFGS) method [1]. An example taken from the literature demonstrates that this technique yields sharp filter having reduced magnitude error and total amount of nonzero coefficients.