A/B Testing: How to Run Tests You Can Trust (With Sample Size Guide)

Should the subject line say “Your cart is waiting” or “Still thinking it over?” You could argue about it for an hour, or you could send each to half the list and look at what happened. That is an A/B test. It replaces opinion with evidence, which is why it sits at the centre of any serious email, push and journey programme. This guide explains what A/B testing is, how to run a test that you can trust, how to size it, which things are worth testing, and the errors that produce false winners.

What is A/B testing?

A/B testing is a controlled experiment in which two versions of something, A and B, are shown to similar groups of people at the same time, and the results are compared to see which version performs better on a chosen measure. Group A usually sees the existing version, called the control. Group B sees the change. Because people are split at random, differences in results can be credited to the change rather than to who happened to be in each group.

What you can A/B test in messaging

How to run an A/B test, step by step

1. Write a hypothesis. “Adding the product name to the subject line will increase click rate because it is more specific.” A hypothesis forces you to say what you expect and why.
2. Change one thing. If you change the subject, the image and the offer at once, you will not know which one mattered.
3. Pick one primary metric. Choose it before you start. Clicks or conversions beat opens, because Apple Mail Privacy Protection inflates opens.
4. Work out the sample size. Decide how many people each group needs so that a real difference can be detected. More below.
5. Split randomly and send at the same time. Do not let one group get a weekday and the other a weekend.
6. Wait for the planned finish. Resist looking early and stopping when one side leads.
7. Analyse and decide. If the difference is statistically significant and large enough to matter, roll out the winner. Otherwise treat it as a tie.
8. Record what you learned. Keep a log of tests, results and the reason. Patterns across tests are where the real insight is.

How big should the test be?

Small tests produce noisy results. If you send each version to 200 people and one gets 11 clicks while the other gets 9, that is not evidence of anything. The size you need depends on your baseline rate and the smallest improvement you care about. The smaller the lift you want to detect, the more people you need. The idea behind it is statistical significance: a measure of how unlikely it is that the difference you see is just chance. Teams commonly accept a 95 percent confidence level, meaning a 5 percent risk of calling a winner that is not real.

from math import sqrt, ceil

def sample_per_group(p_base, p_target, z_alpha=1.96, z_beta=0.84):
    """Approximate people needed per variant (95% confidence, 80% power)."""
    p_bar = (p_base + p_target) / 2
    num = (z_alpha * sqrt(2 * p_bar * (1 - p_bar))
           + z_beta * sqrt(p_base * (1 - p_base) + p_target * (1 - p_target))) ** 2
    return ceil(num / (p_target - p_base) ** 2)

# Example: baseline click rate 3%, hope to reach 3.6%
print(sample_per_group(0.03, 0.036))  # roughly 14,000 per group

That example shows why small lists struggle to test small differences. If your list is a few thousand people, test bigger swings (a different offer, not a different comma), or test across several sends and combine the results.

Common A/B testing mistakes

• Testing too many things. One variable per test.
• Stopping early. A lead on day one often vanishes by day three.
• Using open rate as the winner. Pre-loading by mail apps distorts it.
• Ignoring seasonality. A test during a festival week may not apply in a normal week.
• Testing trivia. Button colours rarely matter as much as the offer and the timing.
• Running overlapping tests on the same people. The effects mix.
• Not recording results. You will repeat old tests.

How to read the results

After a test ends, look at four things. First, the size of the difference: a 0.1 point gain may be real but not worth a change. Second, the confidence that it is not chance. Third, the segments: did the winner win for everyone, or only for new customers? Fourth, the secondary effects, such as unsubscribes, which can rise even when clicks do. If the result is a tie, that is useful too: choose the simpler or cheaper version and move on to a bigger idea.

Beware of running many small tests and celebrating the one that happened to win. If you check twenty variants at a 95 percent confidence level, you should expect about one false winner by luck alone. This is the problem of multiple comparisons. The honest fixes are to decide the primary metric in advance, limit the number of variants and confirm a surprising win by running it again. A sample size calculation before each test keeps you honest about what is possible with your list.

What about multivariate tests?

Multivariate testing changes several elements at once, for example headline, image and button, and measures every combination. It can show how elements interact, but it needs far more traffic than a simple A/B test. For most messaging programmes, a sequence of single-variable tests is more practical.

Test ideas worth your time

Start where the potential gain is largest. In rough order of impact: the offer, the audience (to whom you send), the timing and channel order, the subject line and preview, and last the design details. For an abandoned cart journey, for example, test the first delay (one hour against four) before you test the button colour. See our cart recovery guide for the structure of that sequence.

A/B testing in Flowfox

In campaigns you can create variants of subject, content and send time, choose the split and let the system send the winner to the rest of the audience after the test group has finished. In journeys a split step sends part of the audience down each path, for example WhatsApp first and email first, and a goal measures which converts more. Analytics compares the variants by clicks, conversions and revenue. FoxAI can draft alternative subject lines to test, and the send a campaign guide covers the first send.

A/B tests vs holdout groups

Use both. A/B tests improve the message. Holdouts prove that the programme is worth running at all.

Most tests do not produce a winner. That is not failure. It is information about what does not matter.

Testing works best on a base of good data and clear audiences. Read about customer segmentation, email marketing and marketing automation, and see more on the blog.