How to run a post hoc test
- Copy a group name column and a measurement column together with the header.
- Select the response and group columns and choose the post hoc method. For Dunnett, also choose the control group.
- Press Analyze and check the 'Significant' rows of the results table and the CLD table.
Worked example with sample data
The sample data are 8 adhesion measurements (N) for each of 4 coating conditions (Condition 1 to 4). The Coating condition column is the group column and the Adhesion column the response column.
Tukey HSD shows Condition 1-3 (mean difference 4.925, p=0.0007), Condition 1-4 (6.675, p=0, i.e. below 0.00005) and Condition 2-4 (4.138, p=0.0043) as significant. Tukey's mean difference is 'group 2 - group 1', so a positive value means the second group has the higher mean.
The compact letter display gives Condition 4 c, Condition 3 bc, Condition 2 ab and Condition 1 a. Neighboring conditions that share a letter do not differ significantly, while pairs with no shared letter - 1 and 4, 1 and 3, 2 and 4 - differ significantly.
In the mean difference chart with confidence intervals, the three pairs whose intervals do not cross the dashed line at 0 are the significant ones. Choosing Dunnett with Condition 1 as the control shows Condition 3 (p=0.0003381) and Condition 4 (p=4.468e-06) as significant and Condition 2 (p=0.07442) as not.
Choosing a method
Tukey HSD is the standard way to compare all pairs (assumes equal variances), and Dunnett compares only the control with the others, which gives more power. Games-Howell does not assume equal variances, and Bonferroni/Holm correct Welch t test p-values (Holm is less conservative).
Tukey's mean difference is 'group 2 - group 1'; the other methods use 'group 1 - group 2' (Dunnett: group - control).
Compact letter display
Groups that do not differ significantly share a letter. Two groups that share no letter differ significantly, so one table summarizes all the comparisons.
What the result shows
The post hoc results table, mean differences with confidence intervals (dashed line at 0), the compact letter display, and a group mean plot.
How to read the post hoc test result
- Pairs whose adjusted p-value is below the significance level (Significant) have different means. The interpretation tells you which group is higher and by how much.
- In the mean difference chart, pairs whose interval does not cross 0 are significant.
- The intervals in the group mean plot are each group's own confidence interval, so a pair can be significant in the post hoc test even if the intervals overlap.
Frequently asked questions
Can I run post hoc tests without an ANOVA?
Yes. When there are many pairs, though, it is common to check the overall difference with ANOVA first.
Does the Dunnett p-value change every run?
No. The random seed of the multivariate t integration is fixed, so the same data always give the same value.
Should I use Tukey or Bonferroni?
When comparing all pairs, Tukey usually has more power. When only a few pairs are chosen in advance, Bonferroni or Holm fit better.