AI Native · interactive
A/B testing an AI feature
You change the prompt and want to know if users like it more. In truth, version B gets thumbs-up 65.0% of the time vs 62.0% for A, a small real improvement. How many users before the data shows it?
Prompt A · 50 users
76.0%
thumbs up
Prompt B · 50 users
58.0%
thumbs up
Difference B − A, with 95% interval
Purple tick: the true difference (+3 points), which you never get to see in real life.
Not significant · z = -1.91
With only 50 users per side, A looks better, by chance. The interval is wide and includes 0, so this result means nothing yet.
1 2 3 Detecting a 3-point change needs thousands of users per side. Decide the sample size up front and don't stop the moment it looks good.
Peeking inflates false positives.
4
In production
Tools and technologies
Experiment platforms handle the statistics for you.
Not tied to one language — use these names when you search docs or design a system.