Distortion and Information

A pair random process (X; Y) = {X n , Y n } can be generated by a source and a channel, a source and a code, a combination of a source, channel, encoder, and decoder, or two sources and a coupling. We have developed in some detail the properties of two quantities characterizing relations between the two components of a pair process: the average distortion between the components and their mutual information rate. In this chapter relations are developed between distortion and rate, where rate is measured by mutual information. The primary results concern the Shannon distortion-rate function and its dual, the Shannon rate-distortion function, which will be seen to provide bounds on the relationships between distortion and entropy and to characterize the optimal performance in source coding and rate-constrained simulation systems.