You ran a clean A/B test, got a confident winner, and shipped it — except your traffic split was never actually 50/50. That's called sample ratio mismatch, and it can quietly invalidate results that looked statistically sound.

This post explains what causes it and how to catch it before it costs you.

What Sample Ratio Mismatch (SRM) Actually Is

If you set up a 50/50 split but your data shows 55/45 (or worse), something is broken in how traffic is being assigned — not in the page itself.

  • It means your two groups aren't actually comparable anymore.
  • Any "winner" from a mismatched test may just reflect a biased sample, not a real effect.
  • SRM is detectable with a simple chi-squared test on your raw visitor counts.

Common Causes of SRM

  • Bots and crawlers disproportionately hitting one variant.
  • Caching issues where one variant loads faster and gets counted more often.
  • Redirect-based tests where the redirect itself fails intermittently for one group.
  • Ad blockers interfering with the testing script on one variation but not the other.

How to Check for It

  1. Before analyzing conversion rate, check the raw visitor count split between variants.
  2. Run a quick chi-squared test — most A/B testing tools automatically flag this.
  3. If mismatched, pause the test and investigate implementation before trusting any results.

Prevention Tips

  • Test your implementation on multiple browsers and devices before launch.
  • Exclude known bot traffic at the analytics layer, not just at the reporting stage.
  • Avoid client-side redirects for test variants where possible — they're the most common SRM source.

Conclusion

A confident-looking result built on a mismatched sample is worse than no result at all. Always check your traffic split before trusting a winner, and fix the implementation issue before running the test again.