Post Hoc Tests (Tukey HSD, Dunnett, Games-Howell)

Once ANOVA says 'there is a difference', the next question is which groups differ. A pairwise comparison table, confidence intervals of the mean differences and a CLD table that groups means by shared letters summarize the result at a glance.

1Data Table

Paste a table copied from Excel into a cell (Ctrl+V). Click a column header (A, B…) before pasting to use the first line as column names. Double-click a header to rename it, right-click to insert/delete rows and columns, and formulas such as =A1*2 work too.

2Post Hoc Tests Options

How to run a post hoc test

  1. Copy a group name column and a measurement column together with the header.
  2. Select the response and group columns and choose the post hoc method. For Dunnett, also choose the control group.
  3. Press Analyze and check the 'Significant' rows of the results table and the CLD table.

Worked example with sample data

The sample data are 8 adhesion measurements (N) for each of 4 coating conditions (Condition 1 to 4). The Coating condition column is the group column and the Adhesion column the response column.

Mean differences with confidence intervals from the Tukey HSD post hoc test of coating condition 1 to 4 adhesion
Mean differences with confidence intervals from the Tukey HSD post hoc test of coating condition 1 to 4 adhesion

Tukey HSD shows Condition 1-3 (mean difference 4.925, p=0.0007), Condition 1-4 (6.675, p=0, i.e. below 0.00005) and Condition 2-4 (4.138, p=0.0043) as significant. Tukey's mean difference is 'group 2 - group 1', so a positive value means the second group has the higher mean.

The compact letter display gives Condition 4 c, Condition 3 bc, Condition 2 ab and Condition 1 a. Neighboring conditions that share a letter do not differ significantly, while pairs with no shared letter - 1 and 4, 1 and 3, 2 and 4 - differ significantly.

In the mean difference chart with confidence intervals, the three pairs whose intervals do not cross the dashed line at 0 are the significant ones. Choosing Dunnett with Condition 1 as the control shows Condition 3 (p=0.0003381) and Condition 4 (p=4.468e-06) as significant and Condition 2 (p=0.07442) as not.

Choosing a method

Tukey HSD is the standard way to compare all pairs (assumes equal variances), and Dunnett compares only the control with the others, which gives more power. Games-Howell does not assume equal variances, and Bonferroni/Holm correct Welch t test p-values (Holm is less conservative).

Tukey's mean difference is 'group 2 - group 1'; the other methods use 'group 1 - group 2' (Dunnett: group - control).

Compact letter display

Groups that do not differ significantly share a letter. Two groups that share no letter differ significantly, so one table summarizes all the comparisons.

What the result shows

The post hoc results table, mean differences with confidence intervals (dashed line at 0), the compact letter display, and a group mean plot.

How to read the post hoc test result

  • Pairs whose adjusted p-value is below the significance level (Significant) have different means. The interpretation tells you which group is higher and by how much.
  • In the mean difference chart, pairs whose interval does not cross 0 are significant.
  • The intervals in the group mean plot are each group's own confidence interval, so a pair can be significant in the post hoc test even if the intervals overlap.

Frequently asked questions

Can I run post hoc tests without an ANOVA?

Yes. When there are many pairs, though, it is common to check the overall difference with ANOVA first.

Does the Dunnett p-value change every run?

No. The random seed of the multivariate t integration is fixed, so the same data always give the same value.

Should I use Tukey or Bonferroni?

When comparing all pairs, Tukey usually has more power. When only a few pairs are chosen in advance, Bonferroni or Holm fit better.

Other chart tools