A/B Testing Email Subject Lines: Right vs. Wrong

Visisto is the complete website growth system — and A/B testing is built into every email broadcast and sequence, because email tests are the fastest way to compound your marketing results.

70% of email A/B tests produce results that aren't statistically significant. Not because the tests were run wrong, but because the sample size was too small. A store with 800 subscribers runs a 50/50 split, gets 30% open rate on Variant A and 27% on Variant B, and concludes Variant A wins. It doesn't. With 400 recipients per variant, that 3-point difference could be explained by random variation.

The sample size problem

To detect a 5-percentage-point difference in open rate at 95% confidence, you need approximately 500 subscribers per variant. To detect a 2-point difference, you need 3,000 per variant. Most smaller stores don't have the list size to run meaningful open rate tests — and that's fine, as long as you know it.

For lists under 2,000: don't run subject line A/B tests. Instead, manually try different approaches across sequential sends and look for patterns over time. For lists over 6,000: A/B testing becomes genuinely informative.

What to test (in priority order)

• Presence of a specific number: 'Your order is ready' vs. '1 item waiting for you'
• Length: under 35 characters vs. over 50 characters
• Question vs. statement: 'Ready to save 15%?' vs. 'Your 15% discount is ready'
• Personalisation: first name in subject line vs. none
• Urgency framing: 'Expires tonight' vs. 'Sale now on'

The right test structure

Don't do 50/50 splits. Use 40/40/20: 40% gets Variant A, 40% gets Variant B, 20% is held back. After 4 hours (when most open activity occurs), send the winner to the remaining 20%. This gives you a valid test while maximising sends to the winning variant.

Test one variable at a time. If you change the subject line length AND add a number AND change the tone, you don't know which change drove the result.

What Visisto's testing shows at scale

Across 180,000 A/B tested broadcasts in Q1 2026, the highest-impact single variable was presence of a specific number in the subject line — associated with a 4.2-percentage-point average open rate lift. Second: question format (+2.8pp). Third: first name personalisation (+1.9pp).

If you want A/B testing built into every broadcast with automatic winner selection, Visisto is free to start — no credit card: visisto.com