"Run it until it's significant" is the most common — and most misleading — advice in A/B testing. Significance can appear and disappear within the same week if you stop too early.

Here's a practical framework for deciding test duration before you even launch.

Why "Just Wait for Significance" Fails

Statistical significance is a snapshot, not a guarantee. Checking daily and stopping the moment you see 95% confidence is a fast way to ship false winners.

  • Weekday and weekend behavior can differ dramatically for e-commerce.
  • Payday cycles and promotional periods distort short test windows.
  • A test that "wins" on day 3 can flip by day 10.

The Two Numbers You Need Before Starting

  1. Minimum detectable effect (MDE): the smallest lift you actually care about detecting.
  2. Required sample size: calculated from your current conversion rate and MDE using an A/B test calculator.

Only once you know these can you estimate an honest test duration based on your daily traffic.

The Full-Cycle Rule

  • Run tests in multiples of full 7-day weeks — never stop mid-week.
  • For stores with monthly promotional cycles, avoid tests that span a major sale unless that's specifically what you're testing.
  • If your required sample size takes longer than 4–6 weeks to reach, consider testing a bigger, bolder change instead of a subtle one.

Red Flags That You're Stopping Too Early

  • You're checking results daily and feel the urge to stop the moment it looks good.
  • Your sample size is far below what the calculator recommended.
  • The result flips direction between two consecutive days.

Conclusion

Test duration isn't a feeling — it's math. Calculate your required sample size before launching, commit to full-week cycles, and resist the urge to call a winner the moment the numbers look good.