A study on quantization effects of DCT based compression

Quantization in discrete cosine transform (DCT) domain is a widely used lossy compression technique in international multimedia compression standards from audio (e.g., MP3) to image (e.g., JPEG) to video (e.g., H.264). Unlike many degradation sources, like sensor noises, quantization errors of DCT coefficients are signal dependent and difficult to isolate and remove, causing serious artifacts in decompressed and post-processed signals. In this research, quantization errors in the DCT domain are analytically assessed. Our analysis exposes complex behaviors of the DCT quantization errors after being mapped back into the temporal or spatial domain. These behaviors are highly sensitive to quantization precision, the amplitude and phase of the input signal. Based on these observations, we develop a DCT-domain error model to predict and quantify the quantization effects in cases where artifacts are most perceivable to humans, and offer some insights into possible strategies for further suppressing compression noises.