Cramér–von Mises criterion

Summarize Timeline Fact Check

Let ${\displaystyle x_{1},x_{2},\ldots ,x_{n}}$ be the observed values, in increasing order. Then the test statistic is^{[3]: 1153 [5]}

$${\displaystyle T=n\omega ^{2}={\frac {1}{12n}}+\sum _{i=1}^{n}\left[{\frac {2i-1}{2n}}-F^{*}(x_{i})\right]^{2}.}$$

If this value is larger than the tabulated value, then the hypothesis that the data came from the distribution ${\displaystyle F^{*}}$ can be rejected.

Watson test

A modified version of the Cramér–von Mises test is the Watson test^[6] which uses the statistic U², where^[5]

$${\displaystyle U^{2}=T-n({\bar {F}}-{\tfrac {1}{2}})^{2},}$$

where $${\displaystyle {\bar {F}}={\frac {1}{n}}\sum _{i=1}^{n}F^{*}(x_{i}).}$$

Summarize Timeline Top Qs Fact Check

Let ${\displaystyle x_{1},x_{2},\ldots ,x_{n}}$ and ${\displaystyle y_{1},y_{2},\ldots ,y_{m}}$ be the observed values in the first and second sample respectively, in increasing order. Within the combined sample of size ${\displaystyle n+m}$, let ${\displaystyle r_{1},r_{2},\ldots ,r_{n}}$ be the ranks of the xs in the combined sample, and let ${\displaystyle s_{1},s_{2},\ldots ,s_{m}}$ be the ranks of the ys in the combined sample. Anderson^[3]: 1149 shows that

$${\displaystyle T={\frac {nm}{n+m}}\omega ^{2}={\frac {U}{nm(n+m)}}-{\frac {4mn-1}{6(m+n)}}}$$

where U is defined as

$${\displaystyle U=n\sum _{i=1}^{n}(r_{i}-i)^{2}+m\sum _{j=1}^{m}(s_{j}-j)^{2}}$$

If the value of T is larger than the tabulated values,^{[3]: 1154–1159} the hypothesis that the two samples come from the same distribution can be rejected. (Some books^[specify] give critical values for U, which is more convenient, as it avoids the need to compute T via the expression above. The conclusion will be the same.)

The above assumes there are no duplicates in the ${\displaystyle x}$, ${\displaystyle y}$, and ${\displaystyle r}$ sequences. So ${\displaystyle x_{i}}$ is unique, and its rank is ${\displaystyle i}$ in the sorted list ${\displaystyle x_{1},\ldots ,x_{n}}$. If there are duplicates, and ${\displaystyle x_{i}}$ through ${\displaystyle x_{j}}$ are a run of identical values in the sorted list, then one common approach is the midrank^[7] method: assign each duplicate a "rank" of ${\displaystyle (i+j)/2}$. In the above equations, in the expressions ${\displaystyle (r_{i}-i)^{2}}$ and ${\displaystyle (s_{j}-j)^{2}}$, duplicates can modify all four variables ${\displaystyle r_{i}}$, ${\displaystyle i}$, ${\displaystyle s_{j}}$, and ${\displaystyle j}$.

Summarize Timeline Top Qs Fact Check

For two distributions on the real line with cumulative distribution functions ${\displaystyle F}$ and ${\displaystyle G}$ and finite first moment, the Cramér distance is

${\displaystyle \ell _{2}(F,G)=\left[\int _{-\infty }^{\infty }{\bigl (}F(x)-G(x){\bigr )}^{2}dx\right]^{1/2},}$

a metric on the space of such distributions.^[8] Note that some sources define the Cramér distance as ${\displaystyle \ell _{2}^{2}}$, but this fails the triangle inequality and so cannot be properly defined as a distance. The Cramér distance is the one-dimensional case of the energy distance via the relationship ${\displaystyle {\sqrt {2}}\ell _{2}=D}$,^[9] and when ${\displaystyle G}$ represents a single observation ${\displaystyle y}$ with cumulative distribution ${\displaystyle G(x)=\mathbf {1} \{x\geq y\}}$, ${\displaystyle \ell _{2}^{2}(F,G)}$ is equivalent to the continuous ranked probability score, a strictly proper scoring rule.^[10]

Under the probability integral transform (PIT), the plot of the empirical distribution of the transformed values ${\displaystyle F^{*}(x_{1}),\ldots ,F^{*}(x_{n})}$ and the uniform distribution on ${\displaystyle [0,1]}$ creates a PIT reliability diagram. The Cramér distance ${\displaystyle \ell _{2}}$ between these two distributions equals ${\displaystyle \omega }$, the square root of the criterion, and serves as a numerical score of the calibration error of ${\displaystyle F^{*}}$. This may also be referred to as the Root Mean Square Calibration Error (RMSCE).

For a deterministic (point) forecast at ${\displaystyle \mu }$, the PIT degenerates to a Bernoulli random variable on ${\displaystyle \{0,1\}}$ with success probability ${\displaystyle p=\Pr(y>\mu )}$, so in the population limit the Cramér distance between the PIT CDF and the uniform distribution evaluates in closed form to

${\displaystyle \ell _{2}=\left[\int _{0}^{1}{\bigl (}(1-p)-x{\bigr )}^{2}\,dx\right]^{1/2}={\sqrt {p^{2}-p+{\tfrac {1}{3}}}}.}$

This quantity is minimized at ${\displaystyle p={\tfrac {1}{2}}}$ (the unbiased case) with value ${\displaystyle {\tfrac {1}{\sqrt {12}}}\approx 0.2887}$, establishing a calibration-error floor that no point forecast can fall below regardless of how accurate its central value is. In contrast, a well-calibrated probabilistic forecast can approach 0. Similarly, this quantity is maximized at the bias extremes ${\displaystyle p\in \{0,1\}}$ with value ${\displaystyle {\tfrac {1}{\sqrt {3}}}\approx 0.5774}$.