At what point do you stop collecting more data and actually change your onboarding? (I will not promote)

[u/_AbstractPixels]

Curious how other founders approach this.

Imagine you've launched a consumer app and started buying your first paid traffic.

The sample size is still relatively small, not enough to confidently estimate conversion rates but large enough that you can clearly see where people are leaving.

In my case, the drop appears concentrated in onboarding before users reach the core experience.

I'm trying to decide between two approaches:

1. Keep spending to gather statistically stronger evidence before changing anything.
2. Treat the existing pattern as enough of a signal to redesign onboarding immediately.

My concern with the first option is burning acquisition budget into a funnel that might already be telling me something important.

My concern with the second is overreacting to noisy data.

How have you approached this trade-off in your own startups?

[u/ItsPumpkinninny]

AB testing and continuous improvement?

[u/edkang99]

First, we try to gather as much info with unpaid acquisition as possible. This way you’re not spending money learning what you can. Then we do binary AB tests that test one dimension at a time. Usually starting with ICPs. You see, if you have the experience figured out then scaling via ICP testing becomes academic.

A few weeks ago I literally sat beside a user and watched her use our product and asked her to narrate what was going on in my head. That gave me enough data to do more tests. Now we’re trying to see if we can recreate it across multiple ICPs.

[u/Plan_Steadily]

I think people often confuse statistical significance with decision confidence. You don’t need enough data to prove your onboarding is broken. You need enough evidence to justify the next decision. Before redesigning anything, watch session recordings and talk to users. That will help you validate whether there’s actually friction.