Information theory and measure theory

Many of the concepts in information theory have separate definitions and formulas for continuous and discrete cases. For example, entropy ${\displaystyle \mathrm {H} (X)}$ is usually defined for discrete random variables, whereas for continuous random variables the related concept of differential entropy, written ${\displaystyle h(X)}$, is used (see Cover and Thomas, 2006, chapter 8). Both these concepts are mathematical expectations, but the expectation is defined with an integral for the continuous case, and a sum for the discrete case.

These separate definitions can be more closely related in terms of measure theory. For discrete random variables, probability mass functions can be considered density functions with respect to the counting measure. Thinking of both the integral and the sum as integration on a measure space allows for a unified treatment.

Consider the formula for the differential entropy of a continuous random variable ${\displaystyle X}$ with range ${\displaystyle \mathbb {R} }$ and probability density function ${\displaystyle f(x)}$:

${\displaystyle h(X)=-\int _{\mathbb {R} }f(x)\log f(x)\,dx.}$

This can usually be interpreted as the following Riemann–Stieltjes integral:

${\displaystyle h(X)=-\int _{\mathbb {R} }f(x)\log f(x)\,d\mu (x),}$

where ${\displaystyle \mu }$ is the Lebesgue measure.

If instead, ${\displaystyle X}$ is discrete, with range ${\displaystyle \Omega }$ a finite set, ${\displaystyle f}$ is a probability mass function on ${\displaystyle \Omega }$, and ${\displaystyle \nu }$ is the counting measure on ${\displaystyle \Omega }$, we can write:

${\displaystyle \mathrm {H} (X)=-\sum _{x\in \Omega }f(x)\log f(x)=-\int _{\Omega }f(x)\log f(x)\,d\nu (x).}$

The integral expression, and the general concept, are identical in the continuous case; the only difference is the measure used. In both cases the probability density function ${\displaystyle f}$ is the Radon–Nikodym derivative of the probability measure with respect to the measure against which the integral is taken.

If ${\displaystyle P}$ is the probability measure induced by ${\displaystyle X}$, then the integral can also be taken directly with respect to ${\displaystyle P}$:

${\displaystyle h(X)=-\int _{\Omega }\log {\frac {\mathrm {d} P}{\mathrm {d} \mu }}\,dP,}$

If instead of the underlying measure μ we take another probability measure ${\displaystyle Q}$, we are led to the Kullback–Leibler divergence: let ${\displaystyle P}$ and ${\displaystyle Q}$ be probability measures over the same space. Then if ${\displaystyle P}$ is absolutely continuous with respect to ${\displaystyle Q}$, written ${\displaystyle P\ll Q,}$ the Radon–Nikodym derivative ${\displaystyle {\frac {\mathrm {d} P}{\mathrm {d} Q}}}$ exists and the Kullback–Leibler divergence can be expressed in its full generality:

${\displaystyle D_{\operatorname {KL} }(P\|Q)=\int _{\operatorname {supp} P}{\frac {\mathrm {d} P}{\mathrm {d} Q}}\log {\frac {\mathrm {d} P}{\mathrm {d} Q}}\,dQ=\int _{\operatorname {supp} P}\log {\frac {\mathrm {d} P}{\mathrm {d} Q}}\,dP,}$

where the integral runs over the support of ${\displaystyle P.}$ Note that we have dropped the negative sign: the Kullback–Leibler divergence is always non-negative due to Gibbs' inequality.

Summarize Timeline Top Qs Fact Check

There is an analogy between Shannon's basic "measures" of the information content of random variables and a measure over sets. Namely the joint entropy, conditional entropy, and mutual information can be considered as the measure of a set union, set difference, and set intersection, respectively (Reza pp. 106–108).

If we associate the existence of abstract sets ${\displaystyle {\tilde {X}}}$ and ${\displaystyle {\tilde {Y}}}$ to arbitrary discrete random variables X and Y, somehow representing the information borne by X and Y, respectively, such that:

• ${\displaystyle \mu ({\tilde {X}}\cap {\tilde {Y}})=0}$ whenever X and Y are unconditionally independent, and
• ${\displaystyle {\tilde {X}}={\tilde {Y}}}$ whenever X and Y are such that either one is completely determined by the other (i.e. by a bijection);

where ${\displaystyle \mu }$ is a signed measure over these sets, and we set:

${\displaystyle {\begin{aligned}\mathrm {H} (X)&=\mu ({\tilde {X}}),\\\mathrm {H} (Y)&=\mu ({\tilde {Y}}),\\\mathrm {H} (X,Y)&=\mu ({\tilde {X}}\cup {\tilde {Y}}),\\\mathrm {H} (X\mid Y)&=\mu ({\tilde {X}}\setminus {\tilde {Y}}),\\\operatorname {I} (X;Y)&=\mu ({\tilde {X}}\cap {\tilde {Y}});\end{aligned}}}$

we find that Shannon's "measure" of information content satisfies all the postulates and basic properties of a formal signed measure over sets, as commonly illustrated in an information diagram. This allows the sum of two measures to be written:

${\displaystyle \mu (A)+\mu (B)=\mu (A\cup B)+\mu (A\cap B)}$

and the analog of Bayes' theorem (${\displaystyle \mu (A)+\mu (B\setminus A)=\mu (B)+\mu (A\setminus B)}$) allows the difference of two measures to be written:

${\displaystyle \mu (A)-\mu (B)=\mu (A\setminus B)-\mu (B\setminus A)}$

This can be a handy mnemonic device in some situations, e.g.

${\displaystyle {\begin{aligned}\mathrm {H} (X,Y)&=\mathrm {H} (X)+\mathrm {H} (Y\mid X)&\mu ({\tilde {X}}\cup {\tilde {Y}})&=\mu ({\tilde {X}})+\mu ({\tilde {Y}}\setminus {\tilde {X}})\\\operatorname {I} (X;Y)&=\mathrm {H} (X)-\mathrm {H} (X\mid Y)&\mu ({\tilde {X}}\cap {\tilde {Y}})&=\mu ({\tilde {X}})-\mu ({\tilde {X}}\setminus {\tilde {Y}})\end{aligned}}}$

Note that measures (expectation values of the logarithm) of true probabilities are called "entropy" and generally represented by the letter H, while other measures are often referred to as "information" or "correlation" and generally represented by the letter I. For notational simplicity, the letter I is sometimes used for all measures.