AB Test Statistical Significance for Revenue Metrics
Fifty-seven percent. That's the share of pure A/A tests — tests where there's no real difference between the variants — that declared a false winner at least once, according to Optimizely's own simulations as reported by gironi.it in 2026, when experimenters checked results after every visitor. Not 5%. Fifty-seven. Coin-flip odds you'd have shipped a "winning" change that did nothing, purely because you kept refreshing the dashboard.
The short version: an A/B test reaches trustworthy statistical significance on revenue when you've hit a pre-committed sample size built from your metric's actual standard deviation and your minimum detectable effect — usually five to ten times the traffic of a conversion test — and you don't call it until then. Peek early and you'll win tests that never existed.
Revenue Tests Carry Two Kinds of Variance
Start here, because it's the thing most calculators quietly get wrong. A conversion-rate test measures a binary event: the visitor converted or didn't. Clean. The variance of a proportion is fully determined by the proportion itself, so a sample-size formula can hand you a number without ever seeing your data.
Revenue per visit doesn't work that way. Adobe Target's own documentation puts it plainly: "When using Revenue per Visit (RPV) as a metric, an additional source of variance is added because RPV is the product of revenue per order and conversion rate." You're stacking the noise of whether someone buys on top of the noise of how much they spend. Convert.com's calculator flags the same thing — it "combines two sources of variance (CVR and AOV), so you'll need a bigger sample."
The consequence is subtle but expensive. Conversion-rate variance you can derive from a formula. Revenue variance you have to estimate empirically, from real per-user revenue data, because nobody knows the shape of your order distribution until they look at it. Some businesses have a tidy pile of $80–$120 orders. Others have a long right tail with a $9 impulse buy sitting next to a $40,000 enterprise deal. Those two businesses need wildly different sample sizes for the same lift, and no binary calculator will ever tell them apart.
The Three Numbers That Decide How Long You Run
Before any math, get honest about three inputs. They're the levers, and each one moves your required traffic.
Minimum Detectable Effect (MDE) is the smallest lift you actually care about catching. If a 1% revenue bump wouldn't change a single decision, don't size for it. Smaller MDE means a much bigger sample, and chasing effects too small to matter is how tests run for a quarter and resolve nothing.
Power is your probability of detecting a real effect when one exists, conventionally set at 80%. Lower power means you'll miss real winners. You'll run a genuinely better variant, get an inconclusive result, and kill it.
Standard deviation of the metric is the input a conversion calculator never asks for and can't fake. For revenue it's everything. It captures the whales, the tail, the spread of order sizes — the raw noise you're fighting to see signal through. Feed a revenue test into a binary calculator and you've skipped the one number that determines the answer, which is why ABTestResult.com warns that high-variance continuous metrics behave nothing like conversion tests.
If you only remember one thing from this section: a conversion calculator will confidently give you a wrong number for a revenue test, and it'll look right.
Napkin Math: Sizing Revenue Against Conversion
Let me use deliberately round, made-up numbers so the mechanism is visible — none of these sample figures are benchmarks, they're just arithmetic to show the shape of the problem. Say you run a store with a 5% conversion rate and a $100 average order value, and you want to detect a 10% relative lift.
For the conversion test, you're moving conversion from 5.0% to 5.5%. That's a standard binary sizing problem — for illustration, call it roughly 30,000 visitors per variant at 80% power and 95% confidence. Fine.
Now the revenue test. Revenue per visit averages $5 (5% × $100). But your order sizes aren't uniform. Most orders cluster near $100, a handful land at $600, and one whale drops $12,000. That tail inflates the standard deviation of per-visit revenue far beyond what a $5 average suggests. When you plug the empirical SD into the sizing math, the required sample balloons. ABTestResult.com's rule of thumb captures it directly: "High-variance continuous metrics sometimes require 5 to 10 times more traffic than conversion rate tests."
| Test type | Metric | What drives variance | Sample per variant (illustrative only) |
|---|---|---|---|
| Conversion | CVR 5% → 5.5% | The proportion alone | ~30,000 |
| Revenue | RPV $5, heavy tail | CVR × order-size spread | ~150,000–300,000 |
Those revenue numbers aren't a formula output, and they aren't a source. They're what the 5–10x multiplier looks like once you measure your actual order-size spread and feed the SD in. The range is wide because your tail is what decides it.
I'll take my medicine here. Years ago I sized a checkout redesign off a plain conversion calculator, watched revenue-per-visit tick up, and declared a winner at a sample that turned out to be about a fifth of what the metric needed. We rolled it to 100%. The lift evaporated inside three weeks — it had never been real, just a lucky cluster of big orders during the test window. That mistake is the reason this whole article exists. If you want the broader framing on why revenue-shaped metrics misbehave, our piece on reading revenue retention curves by shape covers the same distribution problem from the cohort side.
The Whale Problem, and How to Trim It
The tail that inflates your sample is usually a few extreme orders. One $40,000 purchase in a test of median $90 buyers can single-handedly swing your variant's average and stall significance for days, because your variance calculation is now dominated by a single human's shopping cart.
The standard fix is winsorization: pick a percentile, cap every value above it at that threshold, re-estimate. Statsig's documentation states its default: "At Statsig, the default percentile for winsorization is 99.9%." So the top 0.1% of revenue values get pulled down to the 99.9th-percentile value rather than deleted. You keep the observation, you just stop letting one outlier set the noise floor for everyone.
Here's the assumption you have to name out loud, because it's where teams lie to themselves. If your business genuinely runs on whales — if the $40k orders are the point and not the noise — then aggressive capping doesn't clean your data, it hides your actual model. Cap cautiously in that case, and be honest that a whale-driven business is simply hard to A/B test on revenue and might need a longer horizon or a proxy metric instead.
The mechanics, once you've decided to trim:
- Choose a percentile (99.9% is a defensible default; go lower only with a reason you can write down).
- Cap every per-user revenue value above it at the threshold value.
- Re-estimate the standard deviation on the capped data.
- Re-run your sample-size math with the new, smaller SD.
That fourth step matters. Capping is only worth doing if you re-size afterward, because the whole payoff is a smaller required sample from lower variance. Skip it and you've just quietly altered your numbers for nothing.
The Peeking Trap
Sizing correctly buys you nothing if you call the test early. Peeking is the act of checking a fixed-horizon experiment repeatedly and stopping the moment it looks significant. It feels like diligence. It's actually the fastest way to manufacture false winners.
The mechanism is well documented. Statsig's own docs put it flatly: "Continuous experiment monitoring (i.e. 'peeking') for the purpose of decision making, however, results in inflated false positive rates." Every time you look, you give randomness another chance to cross the significance line by accident. Look often enough and something eventually will, whether or not there's a real effect underneath.
The damage is quantifiable. Back to those Optimizely A/A simulations reported by gironi.it — tests with no real difference at all:
| Checking frequency | A/A tests declaring a false winner |
|---|---|
| Every visitor | 57% |
| Every 500 visitors | 26% |
| Every 1,000 visitors | 20% |
Read the top row again. Under per-visitor peeking, more than half of tests with nothing to find found something. Even checking every thousand visitors leaves you at 20% — four times the 5% false-positive rate you thought you were running at.
Revenue tests get bitten hardest. Higher variance means noisier interim readings, which means bigger and more tempting fake spikes along the way. A single large order mid-test can shove your variant "ahead" by a margin that looks decisive on a Tuesday and is gone by Thursday. That's exactly the spike I mistook for a winner in my checkout story above. The higher your variance, the more seductive the noise.
Two Honest Fixes: Fixed Horizon or Sequential
There are two legitimate ways out, and they're genuinely different commitments.
| Fixed horizon | Sequential / always-valid | |
|---|---|---|
| What you commit to | Pick a sample size, don't look until you hit it | Look whenever you want, math controls the error |
| Discipline required | High — you must resist peeking | Lower — peeking is the design |
| Sample cost | Baseline | ~20–30% more |
| Failure mode | You peek anyway and inflate false positives | You buy the engine and misuse it |
Sequential testing isn't magic. It pays for the freedom to peek with a larger sample. DRIP's write-up sizes the tradeoff honestly: "The trade-off is modest: you need slightly larger sample sizes, typically 20–30% more than a fixed-horizon test." The method is well grounded. The 2017 KDD paper by Johari and colleagues introduced always-valid p-values and confidence intervals built to survive continuous monitoring, and per the paper it was "deployed by the commercial A/B testing platform Optimizely to communicate experimental results to their customers."
My take, and it's a practitioner's bias: for most teams, discipline beats tooling. If you can commit to a horizon and honor it, fixed-horizon costs you 20–30% less sample and asks only that you don't touch the results early. A sequential engine you don't understand becomes a license to peek that feels rigorous while you cherry-pick the day it crosses the line. Buy sequential when you have a real operational reason to monitor continuously — early-stopping for harm, say — not because peeking is a habit you'd rather not break.
A Runbook You Can Take to the Meeting
Four steps, in order, decided before launch.
First, estimate the standard deviation from real per-user revenue. Not an assumed number, not a conversion proxy. Pull the actual distribution and measure the spread. This is the input that decides everything downstream, and it's the one people skip.
Second, set your MDE and power, then size accordingly. Be ruthless about MDE: the smallest lift you'd act on, not the smallest you could dream of. Expect the revenue answer to land in that 5–10x range above a conversion test, and don't flinch when it does.
Third, decide your capping policy up front and write it down. Pick your winsorization percentile before you see results, so nobody can accuse you of trimming to reach the answer you wanted. Re-estimate SD on the capped data and re-size.
Fourth, commit to a horizon — or adopt a sequential method — before the test goes live. Not on day four when it looks promising. Before.
One note on tooling. Whatever platform you run — Optimizely, Statsig, Adobe Target — the math and the discipline belong to you, not the vendor. The product-side funnel data feeding your revenue metric has to be clean and honest before any of this holds up, and no dashboard will stop you from peeking. That's a decision, not a feature. If you're still assembling the underlying data layer, our guide to choosing a marketing analytics stack for unit economics is a reasonable starting point, and the same rigor applies when you're building a unit-economics dashboard the whole team trusts.
For the deeper statistical grounding, Adobe's sample-size determination docs and the KDD paper on peeking are both worth an hour.
What I'd Tell the CFO
The defensible sentence for the board deck is short: "We sized for a 10% lift, capped whale orders at the 99.9th percentile, locked the horizon before launch, and here's the number." Every clause in that sentence is a decision you made in advance and can show your work on. That's the whole game.
You win the argument the same way you win any test: by not lying to yourself first. The 57% false-winner rate isn't a warning about other people's sloppiness. It's what happens to you the week a big order lands and the dashboard looks great and the meeting is tomorrow. Size it, cap it, lock it, and the number holds up when someone with a CFO title starts asking questions.
FAQ
Why does a revenue A/B test need so much more traffic than a conversion test? Because revenue per visit stacks two sources of variance — whether someone buys and how much they spend — while a conversion test only measures the first. Adobe Target notes RPV is the product of conversion rate and revenue per order, and practitioners report continuous revenue metrics can need 5–10x the traffic, per ABTestResult.com.
Is it ever okay to peek at an A/B test? Only if you're running a sequential/always-valid method designed for it. On a fixed-horizon test, peeking inflates false positives badly — Optimizely's A/A simulations reported by gironi.it hit 57% false winners under per-visitor checking. If you must monitor for harm, use sequential and accept 20–30% more sample.
What percentile should I use for winsorization? Statsig defaults to the 99.9th percentile, which is a reasonable starting point. Go lower only with a documented reason, and if your business genuinely depends on large orders, cap cautiously — over-trimming hides your real model rather than cleaning noise.
Can I just use my normal conversion sample-size calculator for revenue? No. A binary calculator can't take the metric's standard deviation as an input, and for revenue that SD is the number that decides everything. Estimate it from real per-user data and size from there.