[{"data":1,"prerenderedAt":285},["ShallowReactive",2],{"post-\u002Fblog\u002Fselection-bias-in-the-funnel":3,"post-related-\u002Fblog\u002Fselection-bias-in-the-funnel":246,"post-translations-\u002Fblog\u002Fselection-bias-in-the-funnel":277,"post-surround-\u002Fblog\u002Fselection-bias-in-the-funnel":279},{"id":4,"title":5,"body":6,"category":226,"date":227,"description":228,"draft":229,"extension":230,"featured":229,"meta":231,"navigation":232,"path":233,"readingTime":234,"segment":235,"seo":236,"stem":237,"tags":238,"translationKey":243,"updated":244,"__hash__":245},"blog_en\u002Fblog\u002Fselection-bias-in-the-funnel.md","Selection bias: your funnel shows who stayed, not who left",{"type":7,"value":8,"toc":208},"minimark",[9,18,21,26,33,40,43,47,50,57,60,66,70,73,78,85,89,92,96,99,103,106,110,113,117,120,124,127,133,136,143,149,153,156,201,205],[10,11,12,13,17],"p",{},"Every funnel analysis starts with a question almost nobody asks out loud: ",[14,15,16],"strong",{},"who is in this data, and who was left out of it?"," In my view, a good share of the bad product and marketing decisions I've seen live right inside that question, including a few of my own.",[10,19,20],{},"It's called selection bias, and it's treacherous because it doesn't show up on the chart. The chart is right. The numbers are right. The problem is the slice of people who made it there.",[22,23,25],"h2",{"id":24},"what-selection-bias-is","What selection bias is",[10,27,28,29,32],{},"Put simply: selection bias happens when the group you're analyzing ",[14,30,31],{},"doesn't represent the group you want to make a decision about",". You look at one part, draw a conclusion and apply it to the whole.",[10,34,35,36,39],{},"The classic example is the World War II planes. Engineers studied the planes that came back from missions, marked where the bullet holes were and wanted to reinforce those areas. The statistician Abraham Wald pointed out the opposite: the holes showed where a plane ",[14,37,38],{},"could"," be hit and still make it back. The spots that needed armor were exactly the ones with no marks at all, because the planes hit there never came back to be studied.",[10,41,42],{},"Your funnel is full of planes that never came back.",[22,44,46],{"id":45},"ive-fallen-for-it-and-you-probably-have-too","I've fallen for it, and you probably have too",[10,48,49],{},"In my ebook I describe two situations that, looking back, are pure selection bias.",[10,51,52,53,56],{},"The first was in customer interviews. There's a big catch here: the vast majority of customers willing to sit for an interview are ",[14,54,55],{},"mildly satisfied"," customers. The very unhappy ones don't want to talk, and the indifferent ones don't have time. If I had drawn conclusions from the interviews alone, I'd have thought the product was fine and just missing details. That's why I started mixing the conversations with an anonymous form, with open space for criticism, to reach exactly the people who wouldn't show up any other way.",[10,58,59],{},"The second was in the first sales. I went after people I knew, over WhatsApp, presenting the value proposition, and I sold. Great, but I couldn't assume that validated the hypothesis, because part of the purchase was trust in me, not in the product. The group that bought didn't represent the stranger who would arrive through an ad. I only considered the value hypothesis tested once it started selling to people who had never heard of me.",[10,61,62,63],{},"Notice that in neither case was the data wrong. ",[14,64,65],{},"The slice was.",[22,67,69],{"id":68},"where-it-shows-up-in-the-funnel","Where it shows up in the funnel",[10,71,72],{},"A few places where I'm always suspicious:",[74,75,77],"h3",{"id":76},"step-by-step-conversion-rate","Step-by-step conversion rate",[10,79,80,81,84],{},"Step 3's rate is calculated only with those who got past step 2. So when you improve step 2 and bring more people to step 3, step 3 starts receiving ",[14,82,83],{},"different people",", less convinced, who used to drop off earlier. It's natural for step 3's rate to fall, and it doesn't mean it got worse. I've seen teams celebrate one step's improvement and, the following week, open an investigation into the next step's \"drop\".",[74,86,88],{"id":87},"the-feature-that-retains-more","The feature that \"retains more\"",[10,90,91],{},"\"People who use the report retain much more, so let's push the report to everyone.\" Maybe. But whoever went looking for the report on their own was probably already the most engaged user, and would have stayed anyway. The report may be a consequence of engagement, not its cause. Pushing the feature to people who didn't ask for it might change nothing.",[74,93,95],{"id":94},"satisfaction-surveys","Satisfaction surveys",[10,97,98],{},"NPS and in-product surveys only hear from people still in the product. Whoever churned doesn't answer. If you use that to decide what to improve, you're designing the product for those who already stayed.",[74,100,102],{"id":101},"consent-based-measurement-and-blockers","Consent-based measurement and blockers",[10,104,105],{},"People who accept cookies aren't the same as people who decline, and ad-blocker users tend to have a different profile. Your analytics sees part of the audience, and that part isn't random. It's worth knowing the size of what you can't see before you nail down a conclusion.",[74,107,109],{"id":108},"the-channel-that-converts-best","The channel that \"converts best\"",[10,111,112],{},"Branded search almost always shows up as the highest-converting channel. Of course: whoever types your name has already decided. The channel didn't convert that person, it was just on their path. Giving it the credit and cutting the rest can dry up exactly what was getting people to search for your name.",[74,114,116],{"id":115},"recent-cohorts","Recent cohorts",[10,118,119],{},"People who signed up last week haven't had time to buy, come back or cancel yet. Comparing the new cohort with one from three months ago, without matching the window, makes any recent change look worse (or better) than it is.",[22,121,123],{"id":122},"the-mistake-of-picking-where-to-act-by-the-biggest-drop","The mistake of picking where to act by the biggest drop",[10,125,126],{},"Drop is the term we use for the point where a group of users leaves in common. And the natural reflex, when we open a funnel, is to go straight to the step with the biggest drop. Sometimes that's right. In the ebook I show an experiment where attacking the drop at the top of the page improved every step after it.",[10,128,129,130],{},"But before choosing where to act, I try to answer one question: ",[14,131,132],{},"are the people leaving the ones I wanted to keep?",[10,134,135],{},"A step that filters out people who aren't a fit is a healthy step. A big drop on a qualification form may be the funnel working exactly as it should. If you \"fix\" that step, you push people who were never going to buy further down, final conversion falls, the sales team wastes time and the conclusion becomes \"step 4 got worse\".",[10,137,138,139,142],{},"Something I mention in the ebook goes along the same lines: when the ad message is more specific, it tends to bring fewer people to the site, but people more likely to move forward. Looking only at volume, it seems worse. Looking at the whole funnel, it's better. It's the same reasoning in reverse: ",[14,140,141],{},"changing who comes in changes every rate after it",".",[144,145,146],"warning",{},[10,147,148],{},"Be suspicious of every rate improvement that comes with a drop in volume, and every rate drop that comes with a rise in volume. Most of the time, what changed was the group, not the step.",[22,150,152],{"id":151},"how-i-try-to-protect-myself","How I try to protect myself",[10,154,155],{},"There's no way to eliminate selection bias, but you can reduce the damage a lot. What I usually do:",[157,158,159,166,172,178,189,195],"ol",{},[160,161,162,165],"li",{},[14,163,164],{},"I define the population before looking at the number."," Who is this decision about? All visitors? People from ads? Existing customers? Writing that down first keeps me from picking the slice that confirms what I already thought.",[160,167,168,171],{},[14,169,170],{},"I ask who isn't in the data."," Who gave up before the first event, who declined cookies, who didn't answer, who canceled. Sometimes you can't measure it, but you can estimate its size and be careful with the conclusion.",[160,173,174,177],{},[14,175,176],{},"I analyze by entry cohort, not by isolated step."," A group that came in during the same period, from the same source, followed from start to finish. That's what I do in experiments: same source, same window of days, starting from the same initial event.",[160,179,180,183,184,142],{},[14,181,182],{},"I separate correlation from cause with experiments."," If I think a feature or a message causes the result, the honest path is to test with randomized groups, one thing at a time. I wrote about that in ",[185,186,188],"a",{"href":187},"\u002Fblog\u002Fab-testing-with-little-traffic","A\u002FB testing with little traffic",[160,190,191,194],{},[14,192,193],{},"I look at volume and rate together."," Rate without volume misleads, and so does volume without rate.",[160,196,197,200],{},[14,198,199],{},"I listen to the people who don't want to talk."," Anonymous forms, conversations with people who canceled or asked for a refund. They're the hardest to reach and, in my view, the ones who teach the most.",[22,202,204],{"id":203},"summary","Summary",[10,206,207],{},"The funnel shows the behavior of whoever made it there, and whoever makes it there is never a random sample of your audience. Before deciding where to act, ask who is in the data, who was left out and whether the people leaving are the ones you wanted to keep. A big drop isn't always a problem, a falling rate isn't always a decline and the channel that converts most isn't always the one that brought the most. When in doubt, compare similar groups, change one thing at a time and look at the whole picture.",{"title":209,"searchDepth":210,"depth":210,"links":211},"",3,[212,214,215,223,224,225],{"id":24,"depth":213,"text":25},2,{"id":45,"depth":213,"text":46},{"id":68,"depth":213,"text":69,"children":216},[217,218,219,220,221,222],{"id":76,"depth":210,"text":77},{"id":87,"depth":210,"text":88},{"id":94,"depth":210,"text":95},{"id":101,"depth":210,"text":102},{"id":108,"depth":210,"text":109},{"id":115,"depth":210,"text":116},{"id":122,"depth":213,"text":123},{"id":151,"depth":213,"text":152},{"id":203,"depth":213,"text":204},"analytics","2026-08-13","Whoever shows up in your data is never your whole audience. How selection bias distorts funnel analysis and leads teams to act on the wrong step, and what I do to guard against it.",false,"md",{},true,"\u002Fblog\u002Fselection-bias-in-the-funnel",6,"dados",{"title":5,"description":228},"blog\u002Fselection-bias-in-the-funnel",[226,239,240,241,242],"funnel","bias","decision-making","cohort","vies-de-selecao-no-funil",null,"KiYPUJkTuKygmLS7Psgl-o3fjhlnJV83R0ytZn8H3dM",[247,257,267],{"title":248,"description":249,"path":250,"date":251,"segment":235,"category":252,"tags":253,"featured":229,"readingTime":210},"A North Star without vanity: how to choose the metric that really guides the product","Registered users, pageviews and downloads make pretty charts and bad decisions. A guide to choosing the metric that matters and building the tree that connects the team's work to it.","\u002Fblog\u002Fnorth-star-without-vanity","2026-10-29","metricas",[254,255,256,226],"metrics","north-star","product",{"title":258,"description":259,"path":260,"date":261,"segment":235,"category":226,"tags":262,"featured":229,"readingTime":210},"A measurement plan before the first event: clean GA4, with consent","Installing GA4 takes five minutes; trusting the data takes months if you start wrong. How I design the event taxonomy, consent and measurement flow together with the feature.","\u002Fblog\u002Fmeasurement-plan-before-the-first-event","2026-10-22",[263,226,264,265,266],"ga4","privacy","events","measurement",{"title":268,"description":269,"path":187,"date":270,"segment":235,"category":271,"tags":272,"featured":229,"readingTime":210},"A\u002FB testing with little traffic: what to do when statistics won't help you","With a few visits a week, a classic A\u002FB test takes months to tell you anything. How to experiment honestly in small products, without false winners.","2026-09-10","experimentacao",[273,274,275,276],"ab-test","experimentation","statistics","conversion",{"pt-br":278},"\u002Fpt-br\u002Fblog\u002Fvies-de-selecao-no-funil",[280,244],{"title":281,"path":282,"stem":283,"description":284,"children":-1},"Hypothesis before evidence: how to build an action thesis in marketing","\u002Fblog\u002Fhypothesis-before-evidence","blog\u002Fhypothesis-before-evidence","Data without a question confirms anything. Why the hypothesis has to come before the evidence, how to write one you can refute and how to combine several into a thesis that tells you where to invest.",1795051148526]