The famous A/B testing stories come from companies with millions of visits a day, where a 1% change in conversion can be detected in hours. In a new product, with a few hundred visits a week, reality is different: the test never ends, or it ends with a winner that's pure chance.
That doesn't mean giving up on experimenting. It means experimenting differently.
Why the classic test doesn't add up
To detect a difference with confidence, you need a sample size that depends on three things: the current conversion rate, the size of the difference you want to detect and the confidence level.
The rule of thumb: the smaller the difference you want to see, the more visits you need, and the relationship isn't linear. Detecting a small improvement on a low conversion rate can require tens of thousands of visits per variant. If you have a few hundred a week, do the math before you start: often the test would take longer than the life of the page itself.
Strategy 1: test big changes
If you can only detect big differences, only test changes that can produce big differences. Not button colors. Things like:
- A completely different value proposition in the headline;
- A page with visible pricing vs. no pricing;
- A sign-up with three fields vs. one with eight;
- A free trial without a card vs. with a card.
Bold changes produce bigger effects, which show up with smaller samples. And even when they "lose", they teach you more.
Strategy 2: move the metric closer to the top
If the final conversion (subscription) is too rare, measure a more frequent event that demonstrably comes before the conversion. Clicks on the CTA, starting the sign-up, reaching the first value action.
The catch: the intermediate metric needs a real relationship with the outcome. A flashy button can increase clicks and decrease subscriptions, because it attracts the curious.
Strategy 3: swap A/B for before and after, with discipline
When you can't split the traffic, a well-done before and after comparison is better than nothing, as long as you control what you can:
- Compare equivalent periods (same number of weeks, no holidays in between);
- Keep acquisition channels stable (don't change the campaign along with the page);
- Record the exact date of the change and any other relevant event;
- Be suspicious of very large effects: check there isn't another explanation.
Strategy 4: qualitative first
With little traffic, five usability tests are worth more than an inconclusive A/B test. Watching real people try to use the page reveals the big problems (the ones that would justify a test) before you spend weeks waiting for significance.
Session recording tools and heatmaps help too, as long as they're set up respecting the user's consent.
Strategy 5: a Bayesian approach to decide, not to prove
Instead of asking "is the difference statistically significant?", a Bayesian approach answers "what's the probability that B is better than A, and how much do I lose if I choose wrong?". For low-risk, reversible product decisions, accepting a high probability (not a certainty) and moving on is usually the most rational thing to do.
The key point is being honest about the level of evidence: "we have signs that B is better" is different from "we proved B is better".
A checklist before running any test
- A written hypothesis: what changes, why we believe it improves things and which metric will show it.
- A calculated sample size: is the test feasible in the time we have?
- A duration set before starting, in whole weeks.
- Guardrail metrics: what can't get worse (cancellations, support tickets).
- A record of the result, including (especially) the tests that came to nothing.
Summary
With little traffic, the classic A/B test is more a source of self-deception than of learning. Prefer bold changes, more frequent metrics, disciplined comparisons and plenty of qualitative work. And above all, be honest about what the data actually says. A good decision made with imperfect evidence is worth more than a false certainty.
Murilo Anzini
Hands-on writing on engineering, product, data and marketing for people who want to understand, and build, the whole journey of a product.