✦ Day 18 · Inferential Statistics

Permutation-Based
Hypothesis Testing

Replace restrictive parametric assumptions with empirical null distributions generated by shuffling condition labels — providing robust, assumption-free statistical inference.

🎲 Label Shuffling 📊 Empirical Null Distribution 🛡 FWER Control 🐍 mne.stats.permutation_t_test 📖 Cohen ANTS Ch. 33

Traditional parametric tests (like t-tests and ANOVA) assume data are normally distributed and independent. EEG data violate both assumptions: distributions are skewed, and adjacent timepoints and channels are highly correlated. Non-parametric permutation testing constructs the null distribution directly from your own data by randomly shuffling condition labels thousands of times, providing rigorous, assumption-free statistical inference.

1

The Permutation Principle
Null Hypothesis via Label Shuffling

If there is no true difference between Condition A and Condition B (the null hypothesis H0), then the condition labels are arbitrary. Exchanging the labels among trials or subjects should produce test statistics that are just as likely as the original un-shuffled data.

By repeating this shuffling 1,000 to 10,000 times, you build an empirical null distribution. The p-value is simply the proportion of permuted test statistics that are equal to or more extreme than your observed statistic.

Cohen’s intuition (ANTS Ch. 33): Permutation testing is like shuffling a deck of cards. If Condition A and Condition B are red and black cards, shuffling them tests whether a high score in red cards happened by pure chance.

2

Multiple Comparisons & Max-Statistic
Controlling Family-Wise Error Rate

Testing 64 channels across 500 time points yields 32,000 separate statistical tests! At α = 0.05, you would expect 1,600 false positives by pure chance.

Max-statistic correction: On each permutation iteration, store only the maximum test statistic across all channels/times. Comparing your observed statistics against this maximum-value distribution automatically controls the Family-Wise Error Rate (FWER) across the entire multidimensional space.

Python 
import mne
from mne.stats import permutation_t_test

# Data shape: (n_subjects, n_timepoints)
X = condition_A_data - condition_B_data

# Non-parametric 1-sample permutation t-test
t_obs, p_values, H0 = permutation_t_test(
    X,
    n_permutations=5000,
    tail=0,
    n_jobs=-1
)

significant_times = np.where(p_values < 0.05)[0]
print(f"Significant time points: {len(significant_times)}")