Data/Experimentation

A/B testing with little traffic: what to do when statistics won't help you

With a few visits a week, a classic A/B test takes months to tell you anything. How to experiment honestly in small products, without false winners.

Murilo Anzini 3 min read

The famous A/B testing stories come from companies with millions of visits a day, where a 1% change in conversion can be detected in hours. In a new product, with a few hundred visits a week, reality is different: the test never ends, or it ends with a winner that's pure chance.

That doesn't mean giving up on experimenting. It means experimenting differently.

Why the classic test doesn't add up

To detect a difference with confidence, you need a sample size that depends on three things: the current conversion rate, the size of the difference you want to detect and the confidence level.

The rule of thumb: the smaller the difference you want to see, the more visits you need, and the relationship isn't linear. Detecting a small improvement on a low conversion rate can require tens of thousands of visits per variant. If you have a few hundred a week, do the math before you start: often the test would take longer than the life of the page itself.

The most common mistake is checking the test every day and stopping when a winner shows up. With low volume, rates swing a lot at the start, and stopping at the first favorable moment almost guarantees a false positive.

Strategy 1: test big changes

If you can only detect big differences, only test changes that can produce big differences. Not button colors. Things like:

  • A completely different value proposition in the headline;
  • A page with visible pricing vs. no pricing;
  • A sign-up with three fields vs. one with eight;
  • A free trial without a card vs. with a card.

Bold changes produce bigger effects, which show up with smaller samples. And even when they "lose", they teach you more.

Strategy 2: move the metric closer to the top

If the final conversion (subscription) is too rare, measure a more frequent event that demonstrably comes before the conversion. Clicks on the CTA, starting the sign-up, reaching the first value action.

The catch: the intermediate metric needs a real relationship with the outcome. A flashy button can increase clicks and decrease subscriptions, because it attracts the curious.

Strategy 3: swap A/B for before and after, with discipline

When you can't split the traffic, a well-done before and after comparison is better than nothing, as long as you control what you can:

  • Compare equivalent periods (same number of weeks, no holidays in between);
  • Keep acquisition channels stable (don't change the campaign along with the page);
  • Record the exact date of the change and any other relevant event;
  • Be suspicious of very large effects: check there isn't another explanation.

Strategy 4: qualitative first

With little traffic, five usability tests are worth more than an inconclusive A/B test. Watching real people try to use the page reveals the big problems (the ones that would justify a test) before you spend weeks waiting for significance.

Session recording tools and heatmaps help too, as long as they're set up respecting the user's consent.

Strategy 5: a Bayesian approach to decide, not to prove

Instead of asking "is the difference statistically significant?", a Bayesian approach answers "what's the probability that B is better than A, and how much do I lose if I choose wrong?". For low-risk, reversible product decisions, accepting a high probability (not a certainty) and moving on is usually the most rational thing to do.

The key point is being honest about the level of evidence: "we have signs that B is better" is different from "we proved B is better".

A checklist before running any test

  1. A written hypothesis: what changes, why we believe it improves things and which metric will show it.
  2. A calculated sample size: is the test feasible in the time we have?
  3. A duration set before starting, in whole weeks.
  4. Guardrail metrics: what can't get worse (cancellations, support tickets).
  5. A record of the result, including (especially) the tests that came to nothing.

Summary

With little traffic, the classic A/B test is more a source of self-deception than of learning. Prefer bold changes, more frequent metrics, disciplined comparisons and plenty of qualitative work. And above all, be honest about what the data actually says. A good decision made with imperfect evidence is worth more than a false certainty.

#ab-test#experimentation#statistics#conversion

Murilo Anzini

Hands-on writing on engineering, product, data and marketing for people who want to understand, and build, the whole journey of a product.

About me

Free ebook

Criando infoprodutos de valor

The method I use to bring product, marketing and sales together when creating digital products: the way startups test hypotheses, understand needs and build what the market wants to buy.

  • Understand the customer and set the initial hypotheses
  • Build and test a lean MVP
  • Measure progress and validate with data
  • Know when and how to pivot
Download for freePDF · 23 pages · in Portuguese · no sign-up