Dunnett's test

Dunnett's test is performed by computing a Student's t-statistic for each experimental, or treatment, group where the statistic compares the treatment group to a single control group.^[8][9] Since each comparison has the same control in common, the procedure incorporates the dependencies between these comparisons. In particular, the t-statistics are all derived from the same estimate of the error variance which is obtained by pooling the sums of squares for error across all (treatment and control) groups. The formal test statistic for Dunnett's test is either the largest in absolute value of these t-statistics (if a two-tailed test is required), or the most negative or most positive of the t-statistics (if a one-tailed test is required).

In Dunnett's test we can use a common table of critical values, but more flexible options are nowadays readily available in many statistics packages. The critical values for any given percentage point depend on: whether a one- or- two-tailed test is performed; the number of groups being compared; the overall number of trials.

The analysis considers the case where the results of the experiment are numerical, and the experiment is performed to compare p treatments with a control group. The results can be summarized as a set of ${\displaystyle (p+1)}$ calculated means of the sets of observations, ${\displaystyle ({\bar {X_{0}}},...,{\bar {X_{p}}})}$, while ${\displaystyle ({\bar {X_{1}}},...,{\bar {X_{p}}})}$ are referring to the treatment and ${\displaystyle {\bar {X_{0}}}}$ is referring to the control set of observations, and ${\displaystyle s}$ is an independent estimate of the common standard deviation of all ${\displaystyle p+1}$ sets of observations. All ${\displaystyle {\bar {X_{i}}}}$ of the ${\displaystyle p+1}$ sets of observations are assumed to be independently and normally distributed with a common variance ${\displaystyle \sigma ^{2}}$ and means ${\displaystyle \mu _{i}}$. There is also an assumption that there is an available estimate ${\displaystyle s^{2}}$ for ${\displaystyle \sigma ^{2}}$.

Dunnett's test's calculation is a procedure that is based on calculating confidence statements about the true or the expected values of the ${\displaystyle p}$ differences ${\displaystyle {\bar {X_{i}}}-{\bar {X_{0}}}}$, thus the differences between treatment groups' mean and control group's mean. This procedure ensures that the probability of all ${\displaystyle p}$ statements ${\displaystyle {\bar {X_{i}}}-{\bar {X_{0}}}}$ being simultaneously correct is equal to a specified value, ${\displaystyle P}$. When calculating one sided upper (or lower) confidence interval for the true value of the difference between the mean of the treatment and the control group, ${\displaystyle P}$ constitutes the probability that this actual value will be less than the upper (or greater than the lower) limit of that interval. When calculating two-sided confidence interval, ${\displaystyle P}$ constitutes the probability that the true value will be between the upper and the lower limits.

First, we will denote the available N observations by ${\displaystyle X_{ij}}$ when ${\displaystyle i=1,...,p}$ and ${\displaystyle j=1,...,N_{i}}$ and estimate the common variance by, for example: ${\displaystyle s^{2}={\frac {\sum _{i=0}^{p}\sum _{j=1}^{N_{i}}(X_{ij}-{\bar {X_{i}}})^{2}}{n}}}$ when ${\displaystyle {\bar {X_{i}}}}$ is the mean of group ${\displaystyle i}$ and ${\displaystyle N_{i}}$ is the number of observations in group ${\displaystyle i}$, and ${\displaystyle n=\sum _{i=0}^{p}N_{i}-(p+1)}$ degrees of freedom. As mentioned before, we would like to obtain separate confidence limits for each of the differences ${\displaystyle m_{i}-m_{0},(i=1,...,p)}$ such that the probability that all ${\displaystyle p}$ confidence intervals will contain the corresponding ${\displaystyle m_{i}-m_{0}}$ is equal to ${\displaystyle P}$.

We will consider the general case where there are ${\displaystyle p}$ treatment groups and one control group. We will write:

${\displaystyle z_{i}={\cfrac {{\bar {X_{i}}}-{\bar {X_{0}}}-(m_{i}-m_{0})}{\sqrt {{\cfrac {1}{N_{i}}}+{\cfrac {1}{N_{0}}}}}}}$

${\displaystyle D_{i}={\cfrac {{\bar {X_{i}}}-{\bar {X_{0}}}-(m_{i}-m_{0})}{s{\sqrt {{\cfrac {1}{N_{i}}}+{\cfrac {1}{N_{0}}}}}}}}$

we will also write: ${\displaystyle D_{i}={\frac {z_{i}}{s}}}$, which follows the Student's t-statistic distribution with n degrees of freedom. The lower confidence limits with joint confidence coefficient ${\displaystyle P}$ for the ${\displaystyle p}$ treatment effects ${\displaystyle m_{i}-m_{0},(i=1,...,p)}$ will be given by:

${\displaystyle {\bar {X_{i}}}-{\bar {X_{0}}}-d_{i}'s{\sqrt {{\frac {1}{N_{i}}}+{\frac {1}{N_{0}}}}},i=1,...,p}$

and the ${\displaystyle p}$ constants ${\displaystyle d_{i}'}$ are chosen so that ${\displaystyle Prob(t_{1}<d_{1}',...,t_{p}<d_{p}')=P}$. Similarly, the upper limits will be given by:

${\displaystyle {\bar {X_{i}}}-{\bar {X_{0}}}+d_{i}'s{\sqrt {{\frac {1}{N_{i}}}+{\frac {1}{N_{0}}}}},i=1,...,p}$

For bounding ${\displaystyle m_{i}-m_{0}}$ in both directions, the following interval might be taken:

${\displaystyle {\bar {X_{i}}}-{\bar {X_{0}}}\pm d_{i}''s{\sqrt {{\frac {1}{N_{i}}}+{\frac {1}{N_{0}}}}},i=1,...,p}$

when ${\displaystyle d_{i}''}$ are chosen to satisfy ${\displaystyle Prob(|t_{1}|<d_{1}',...,|t_{p}|<d_{p}')=P}$. The solution to those particular values of ${\displaystyle d_{i}''}$ for two sided test and ${\displaystyle d_{i}'}$ for one sided test is given in the tables.^[5] An updated table of critical values was published in 1964.^[6]

The following example was adapted from one given by Villars and was presented in Dunnett's original paper.^[5] The data represent measurements on the breaking strength of fabric treated by three different chemical processes compared with a standard method of manufacture.^[10]

Breaking Strength (lbs)

	Standard	Process 1	Process 2	Process 3
1	55	55	55	50
2	47	64	49	44
3	48	64	52	41
Means	50	61	52	45
Variance	19	27	9	21

Dunnett's Test can be calculated by applying the following steps:

1. Input Data with Means and Variances:

• Collect measurements for each group (standard and treatment processes). See the data in the above table for each group's raw numbers, means, and variances.

2. Calculate Pooled Variance ${\displaystyle s^{2}}$:

• Compute the pooled variance across all groups. E.g.,

${\displaystyle {\frac {55^{2}+47^{2}+48^{2}+55^{2}+\ldots +41^{2}-3(50^{2}+61^{2}+52^{2}+45^{2})}{8}}={\frac {152}{8}}=19}$.

3. Calculate Standard Deviation ${\displaystyle s}$:

• Take the square root of the average variance. E.g.,

${\displaystyle s={\sqrt {19}}=4.36}$.

4. Calculate Standard Error:

• The following formula gives the standard error for the difference of two means. E.g.,

${\displaystyle s{\sqrt {\frac {2}{N}}}=4.36{\sqrt {\frac {2}{3}}}=3.56}$.

5. Determine Critical Value ${\displaystyle t}$:

• Use Dunnett's tables to find ${\displaystyle t}$ for the given degrees of freedom and confidence level. E.g.,

For ${\displaystyle p=3}$ and ${\displaystyle {\text{d.f.}}=8}$:

One-sided: ${\displaystyle t=2.42}$

Two-sided: ${\displaystyle t=2.88}$

6. the quantity which must be added to and/or subtracted from the observed differences between the means to give their confidence limits is denoted as ${\displaystyle A}$ (this was termed "allowance" by Tukey), and can be calculated as follows:

• Multiply ${\displaystyle t}$ by the standard error for the difference of two means. E.g.

One-sided:

${\displaystyle A=2.42\times 3.56=8.61}$

Two-sided:

${\displaystyle A=2.88\times 3.56=10.25}$

7. Compute Confidence Limits:

• Calculate the confidence limits for each process compared to the standard. E.g.,

One-sided Limits:

Process 1:

${\displaystyle 61-50-8.61=2.39\,{\text{lbs}}}$

Process 2:

${\displaystyle 52-50-8.61=-6.61\,{\text{lbs}}}$

Process 3:

${\displaystyle 45-50-8.61=-13.61\,{\text{lbs}}}$

Two-sided Limits:

Process 1:

${\displaystyle 61-50\pm 10.25=[0.75,21.25]\,{\text{lbs}}}$

Process 2:

${\displaystyle 52-50\pm 10.25=[-8.25,12.25]\,{\text{lbs}}}$

Process 3:

${\displaystyle 45-50\pm 10.25=[-15.25,5.25]\,{\text{lbs}}}$

8. Draw Conclusions:

• Based on the computed confidence limits, make conclusions about each process compared to the standard. E.g.,

One-sided:

Process 1: Breaking strength exceeds the standard by at least 2.39 lbs.

Process 2: Breaking strength does not exceed the standard (negative value).

Process 3: Breaking strength does not exceed the standard (negative value).

Two-sided:

Process 1: Breaking strength exceeds the standard by between 0.75 lbs and 21.25 lbs.

Process 2: Breaking strength is between -8.25 lbs and 12.25 lbs (may or may not exceed the standard).

Process 3: Breaking strength is between -15.25 lbs and 5.25 lbs (may or may not exceed the standard).