Data Analysis

Critically interpret the results of an A/B test

Also examine effects, intervals, guardrails, experimental quality, and business significance to avoid looking solely at significance.

Customize this prompt

Complete prompt

You are an expert in experimental science and growth analysis. Please interpret the following A/B test.

Experimental hypothesis: [hypothesis]
Experimental design: [Trials, cycles and samples]
Result data: [Sample size, transformation, mean or statistical results of each group]
Main indicator and guardrail indicator: [indicator]

Output: Whether the experiment answered the original question, absolute and relative effects, confidence interval or uncertainty, statistical significance and business significance, changes in guardrail indicators, abnormal sample proportions, experimental period and risk of novel effects, whether the group analysis is reasonable, and the scope of generalization.

Finally, suggestions and basis are given for continuing to go online, extending the experiment, stopping or redesigning. Don't fake precise significance when raw data are missing.

Example output

Group B has an absolute improvement of 2.1 percentage points and a relative improvement of about 5%. The confidence interval does not span 0, but the true effect is still likely to be small. If the activation rate does not drop within 7 days, the traffic can be gradually expanded; the sample proportion and channel distribution should be checked before going online.

USAGE GUIDE

When to use it

Also examine effects, intervals, guardrails, experimental quality, and business significance to avoid looking solely at significance. Use it as a structured starting point, then review the result against your original material.

How to fill the variables

hypothesis: Add concrete facts, context, and constraints. Mark unknown details instead of guessing.
Trials, cycles and samples: Add concrete facts, context, and constraints. Mark unknown details instead of guessing.
Sample size, transformation, mean or statistical results of each group: Add concrete facts, context, and constraints. Mark unknown details instead of guessing.
indicator: Add concrete facts, context, and constraints. Mark unknown details instead of guessing.

Get better results

Describe the audience and intended decision.
Include source material and hard constraints.
State what the model must not invent.

Before you use the output

Verify facts, numbers, and quotations.
Check that uncertainty is clearly labeled.
Test or review high-impact recommendations.

Editorial note: This template has been structured for practical use. Results vary by model and input; verify important outputs against primary sources.