AI Skills Marketplace / Email & Automation

A/B Significance Validator

The A/B Significance Validator is a free Claude Skill that runs a real two-proportion z-test with a confidence interval and minimum-sample-size warning, catching false-positive "winners" on underpowered A/B tests.

↗ View Source ⬇ Download Full Repo (.zip)
MIT
License
2 functions
Decision branches
sample-size floor warning
Escalation rules
Yes
Multi-tool support
Aug 14, 2026
Last updated
16/16 passing
Tests
README.md View raw ↗

What this is (and isn’t)

A statistics function over the raw counts you supply, not a hosted service or an A/B testing platform integration.

Files

FilePurpose
validator.jstwoProportionZTest(), plus exported erf()/normalCdf() — zero dependencies
test.js16-test suite, including hand-verified z-statistic values against the pooled-variance formula
SKILL.mdFull skill definition, loadable in Claude Code/Desktop

The false-positive trap this catches

A 50% relative lift sounds dramatic. On 20 users per variant, it is statistical noise, not a real effect, this skill says so explicitly (significant: false, with a sample-size warning) instead of letting a raw percentage mislead.

FAQ

Is this skill free to use?

Yes. Free to download and run yourself. MV3 charges $175/hr only for implementation help wiring this into your real ESP or ad-platform reporting.

What statistical test does this run?

A real two-proportion z-test with pooled variance under the null hypothesis, using a standard Abramowitz-Stegun approximation of the normal CDF.

Does this replace a real statistician?

For most marketing A/B decisions (email subject lines, landing pages, ad creative), this is the standard, correct test. For more complex experimental designs, consult a statistician.

What is the minimum-sample-size warning?

A rule-of-thumb floor below which even a "significant" p-value is fragile to noise for typical marketing-scale effect sizes. The skill flags this explicitly rather than silently returning a number.

Does my test data get sent to MV3?

No. The test runs entirely in your own environment.

JR
Jordan Reeves
ABM & Outbound Pipeline, MV3 Marketing