The heavy drop problem
The Army drops trucks out of airplanes. This year, 3 of 12 crashed.
A heavy drop is a vehicle pushed out of a cargo plane under parachutes. It usually works: historically only 1 load in 12 burns in, meaning the chutes fail and the vehicle is destroyed. This year the riggers fielded a new harness, and 3 of the 12 drops burned in. The z-test from intro statistics calls that jump significant (p = 0.018): scrap the harness. Before anyone acts on it, look at the null distribution that p-value came from.
All data on this page are notional, invented for teaching.
The null the z-test imagines, and the real one
Bars: the exact binomial null. Curve: the normal the z-test integrates instead. Orange bars are outcomes at least as extreme as the observed count; the shaded curve area is all it sees of that tail.
Build the null yourself
Where the decision flips
Every (n, observed count) pair at this null rate, colored by what the two tests decide at α = 0.05. In the orange cells the z-test rejects and the exact binomial does not. The dashed line is MA206's validity rule np ≥ 10: the flip zone lives entirely on its wrong side.
◆ THE BLUEPRINT
The z-test is an approximation

$$z = \frac{\hat{p} - p}{\mathrm{SE}} \qquad \mathrm{SE} = \sqrt{\frac{p(1-p)}{n}}$$

where \(p\) is the null value and \(\hat{p}\) is the proportion we observed. The p-value is a normal tail area standing in for a binomial one. The stand-in holds when \(np \ge 10\) and \(n(1-p) \ge 10\); below that the binomial is discrete, floored at zero, and skewed, and the curve's thin tail undercounts extreme outcomes.

The exact answer needs no curve

Under H\(_0\), the count of events in \(n\) independent trials is Binomial\((n, p)\), so the honest p-value is a finite sum: $$P(X \ge x) = \sum_{k=x}^{n} \binom{n}{k} p^k (1-p)^{n-k}$$ The marble cup performs this distribution physically; replicate() performs it ten thousand times; pbinom() sums it exactly. All three agree because they are the same computation at different sizes.

◆ THE BLUEPRINT
The null under your settings
Reproduce this exact screen in R

                    

                    
                  

                

              

              
              

                

                  
◆ THE BLUEPRINT

                  
                

                

                  
What to try

                  
    
                    
  • Set a true effect of 60 to 90 seconds. Both p-values fall together.
    • 
                    
  • Reduce the sample size under heavy skew. Small, heavily skewed samples are the only settings where the two tests disagree.
    • 
                    
  • Increase the shuffles from 999 to 49,999. Only the third decimal of the permutation p-value steadies; the test itself does not change.
    • 
                    
  • Rerun the same settings with different seeds. If changing the seed changes your decision, the data were never strong enough to justify that decision.
    • 
                  

                  
What shuffling cannot fix

                  
The permutation test assumes exchangeability: under the null, any subset of sessions could have received the Design A label. Random assignment guarantees this. Observational labels do not, and shuffling cannot fix a label that was chosen rather than assigned.