AI Skills Marketplace / Analytics

Sample-Ratio-Mismatch (SRM) Auditor

The Sample-Ratio-Mismatch Auditor is a free Claude Skill that checks whether an A/B test's actual traffic split matches its intended assignment ratio using a real chi-square goodness-of-fit test, before you trust any metric result the test produced.

↗ View Source ⬇ Download Full Repo (.zip)
MIT
License
4 functions
Decision branches
strict alpha=0.001 threshold + honest df>1 scope limit
Escalation rules
Yes
Multi-tool support
Aug 17, 2026
Last updated
22/22 passing
Tests
README.md View raw ↗

What this is (and isn’t)

A statistical integrity check, not a metric-significance test. Checking whether a test’s traffic split is even correct is a separate, prerequisite question to whether a metric moved. A test can show a “winning” variant purely because broken randomization skewed how many users landed in each arm.

Files

FilePurpose
srm.jscomputeChiSquareStatistic(), pValueChiSquareDf1(), detectSRM(), detectSRMMultiArm() — zero dependencies
test.js22-test suite, including the p-value function cross-checked against the standard published chi-square df=1 critical value table
SKILL.mdFull skill definition, loadable in Claude Code/Desktop

The two real guardrails

A deliberately strict default significance threshold (alpha 0.001, not the usual 0.05), since under correct random assignment a real ratio mismatch should almost never occur by chance. And an honest scope limit on multi-arm tests: an exact p-value beyond 2 arms requires the chi-square CDF for higher degrees of freedom, which this zero-dependency skill does not implement — it returns the statistic and a note to check a standard critical value table rather than shipping an invented number.

FAQ

Is this skill free to use?

Yes. Free to download and run yourself. MV3 charges $175/hr only for implementation help wiring this into your real experimentation platform's actual assignment data.

How is this different from an A/B significance test?

A significance test checks whether a metric moved. This checks whether the traffic split itself is correct, which is a prerequisite question. Broken randomization can produce a fake metric win even before you look at the metric.

Why is the default threshold so much stricter than a typical 0.05?

Under correct random assignment, a real ratio mismatch should almost never happen by chance. A strict threshold (0.001 by default) avoids false alarms on ordinary sampling noise while still catching genuine instrumentation bugs.

Does this work for tests with more than 2 variants?

detectSRMMultiArm() computes the chi-square statistic for any number of arms, but only returns an exact p-value at 2 arms. Beyond that it honestly returns null and tells you to check a standard chi-square critical value table, rather than approximating a number it cannot verify.

Does my test data get sent to MV3?

No. All calculations run entirely in your own environment.

RB
Ryan Brooks
Technical SEO, MV3 Marketing