[{"data":1,"prerenderedAt":248},["ShallowReactive",2],{"post-\u002Fblog\u002Fab-testing-with-little-traffic":3,"post-related-\u002Fblog\u002Fab-testing-with-little-traffic":218,"post-translations-\u002Fblog\u002Fab-testing-with-little-traffic":240,"post-surround-\u002Fblog\u002Fab-testing-with-little-traffic":242},{"id":4,"title":5,"body":6,"category":199,"date":200,"description":201,"draft":202,"extension":203,"featured":202,"meta":204,"navigation":205,"path":206,"readingTime":188,"segment":207,"seo":208,"stem":209,"tags":210,"translationKey":215,"updated":216,"__hash__":217},"blog_en\u002Fblog\u002Fab-testing-with-little-traffic.md","A\u002FB testing with little traffic: what to do when statistics won't help you",{"type":7,"value":8,"toc":186},"minimark",[9,17,20,25,28,35,45,49,56,72,75,79,86,89,93,100,114,118,125,128,132,135,142,146,179,183],[10,11,12,13],"p",{},"The famous A\u002FB testing stories come from companies with millions of visits a day, where a 1% change in conversion can be detected in hours. In a new product, with a few hundred visits a week, reality is different: ",[14,15,16],"strong",{},"the test never ends, or it ends with a winner that's pure chance.",[10,18,19],{},"That doesn't mean giving up on experimenting. It means experimenting differently.",[21,22,24],"h2",{"id":23},"why-the-classic-test-doesnt-add-up","Why the classic test doesn't add up",[10,26,27],{},"To detect a difference with confidence, you need a sample size that depends on three things: the current conversion rate, the size of the difference you want to detect and the confidence level.",[10,29,30,31,34],{},"The rule of thumb: ",[14,32,33],{},"the smaller the difference you want to see, the more visits you need",", and the relationship isn't linear. Detecting a small improvement on a low conversion rate can require tens of thousands of visits per variant. If you have a few hundred a week, do the math before you start: often the test would take longer than the life of the page itself.",[36,37,38],"warning",{},[10,39,40,41,44],{},"The most common mistake is ",[14,42,43],{},"checking the test every day and stopping when a winner shows up",". With low volume, rates swing a lot at the start, and stopping at the first favorable moment almost guarantees a false positive.",[21,46,48],{"id":47},"strategy-1-test-big-changes","Strategy 1: test big changes",[10,50,51,52,55],{},"If you can only detect big differences, ",[14,53,54],{},"only test changes that can produce big differences",". Not button colors. Things like:",[57,58,59,63,66,69],"ul",{},[60,61,62],"li",{},"A completely different value proposition in the headline;",[60,64,65],{},"A page with visible pricing vs. no pricing;",[60,67,68],{},"A sign-up with three fields vs. one with eight;",[60,70,71],{},"A free trial without a card vs. with a card.",[10,73,74],{},"Bold changes produce bigger effects, which show up with smaller samples. And even when they \"lose\", they teach you more.",[21,76,78],{"id":77},"strategy-2-move-the-metric-closer-to-the-top","Strategy 2: move the metric closer to the top",[10,80,81,82,85],{},"If the final conversion (subscription) is too rare, measure a more frequent event that ",[14,83,84],{},"demonstrably comes before"," the conversion. Clicks on the CTA, starting the sign-up, reaching the first value action.",[10,87,88],{},"The catch: the intermediate metric needs a real relationship with the outcome. A flashy button can increase clicks and decrease subscriptions, because it attracts the curious.",[21,90,92],{"id":91},"strategy-3-swap-ab-for-before-and-after-with-discipline","Strategy 3: swap A\u002FB for before and after, with discipline",[10,94,95,96,99],{},"When you can't split the traffic, a well-done ",[14,97,98],{},"before and after"," comparison is better than nothing, as long as you control what you can:",[57,101,102,105,108,111],{},[60,103,104],{},"Compare equivalent periods (same number of weeks, no holidays in between);",[60,106,107],{},"Keep acquisition channels stable (don't change the campaign along with the page);",[60,109,110],{},"Record the exact date of the change and any other relevant event;",[60,112,113],{},"Be suspicious of very large effects: check there isn't another explanation.",[21,115,117],{"id":116},"strategy-4-qualitative-first","Strategy 4: qualitative first",[10,119,120,121,124],{},"With little traffic, ",[14,122,123],{},"five usability tests are worth more than an inconclusive A\u002FB test",". Watching real people try to use the page reveals the big problems (the ones that would justify a test) before you spend weeks waiting for significance.",[10,126,127],{},"Session recording tools and heatmaps help too, as long as they're set up respecting the user's consent.",[21,129,131],{"id":130},"strategy-5-a-bayesian-approach-to-decide-not-to-prove","Strategy 5: a Bayesian approach to decide, not to prove",[10,133,134],{},"Instead of asking \"is the difference statistically significant?\", a Bayesian approach answers \"what's the probability that B is better than A, and how much do I lose if I choose wrong?\". For low-risk, reversible product decisions, accepting a high probability (not a certainty) and moving on is usually the most rational thing to do.",[10,136,137,138,141],{},"The key point is ",[14,139,140],{},"being honest about the level of evidence",": \"we have signs that B is better\" is different from \"we proved B is better\".",[21,143,145],{"id":144},"a-checklist-before-running-any-test","A checklist before running any test",[147,148,149,155,161,167,173],"ol",{},[60,150,151,154],{},[14,152,153],{},"A written hypothesis",": what changes, why we believe it improves things and which metric will show it.",[60,156,157,160],{},[14,158,159],{},"A calculated sample size",": is the test feasible in the time we have?",[60,162,163,166],{},[14,164,165],{},"A duration set before starting",", in whole weeks.",[60,168,169,172],{},[14,170,171],{},"Guardrail metrics",": what can't get worse (cancellations, support tickets).",[60,174,175,178],{},[14,176,177],{},"A record of the result",", including (especially) the tests that came to nothing.",[21,180,182],{"id":181},"summary","Summary",[10,184,185],{},"With little traffic, the classic A\u002FB test is more a source of self-deception than of learning. Prefer bold changes, more frequent metrics, disciplined comparisons and plenty of qualitative work. And above all, be honest about what the data actually says. A good decision made with imperfect evidence is worth more than a false certainty.",{"title":187,"searchDepth":188,"depth":188,"links":189},"",3,[190,192,193,194,195,196,197,198],{"id":23,"depth":191,"text":24},2,{"id":47,"depth":191,"text":48},{"id":77,"depth":191,"text":78},{"id":91,"depth":191,"text":92},{"id":116,"depth":191,"text":117},{"id":130,"depth":191,"text":131},{"id":144,"depth":191,"text":145},{"id":181,"depth":191,"text":182},"experimentacao","2026-09-10","With a few visits a week, a classic A\u002FB test takes months to tell you anything. How to experiment honestly in small products, without false winners.",false,"md",{},true,"\u002Fblog\u002Fab-testing-with-little-traffic","dados",{"title":5,"description":201},"blog\u002Fab-testing-with-little-traffic",[211,212,213,214],"ab-test","experimentation","statistics","conversion","teste-ab-com-pouco-trafego",null,"ZeFew51OAnpKOyLqgW80tb3AOcYRlm7NBMf0kHnEelU",[219,230],{"title":220,"description":221,"path":222,"date":223,"segment":207,"category":224,"tags":225,"featured":202,"readingTime":188},"A North Star without vanity: how to choose the metric that really guides the product","Registered users, pageviews and downloads make pretty charts and bad decisions. A guide to choosing the metric that matters and building the tree that connects the team's work to it.","\u002Fblog\u002Fnorth-star-without-vanity","2026-10-29","metricas",[226,227,228,229],"metrics","north-star","product","analytics",{"title":231,"description":232,"path":233,"date":234,"segment":207,"category":229,"tags":235,"featured":202,"readingTime":188},"A measurement plan before the first event: clean GA4, with consent","Installing GA4 takes five minutes; trusting the data takes months if you start wrong. How I design the event taxonomy, consent and measurement flow together with the feature.","\u002Fblog\u002Fmeasurement-plan-before-the-first-event","2026-10-22",[236,229,237,238,239],"ga4","privacy","events","measurement",{"pt-br":241},"\u002Fpt-br\u002Fblog\u002Fteste-ab-com-pouco-trafego",[243,216],{"title":244,"path":245,"stem":246,"description":247,"children":-1},"AI in the product without the hype: where an assistant really helps the user","\u002Fblog\u002Fai-in-the-product-without-hype","blog\u002Fai-in-the-product-without-hype","Putting a chat on every screen is not an AI strategy. A way to decide where AI creates real value in the product, how to design the experience and which privacy safeguards belong in from day one.",1795021080189]