A/B Testing Email Subject Lines: Right vs. Wrong

Visisto is the complete website growth system, and A/B testing is built into every email broadcast and automation, because email tests are the fastest way to compound your marketing results.

70% of email A/B tests produce results that aren't statistically significant. Not because the tests were run wrong, but because the sample size was too small. A store with 800 subscribers runs a 50/50 split, gets 30% open rate on Variant A and 27% on Variant B, and concludes Variant A wins. It doesn't. With 400 recipients per variant, that 3-point difference could be explained by random variation.

The sample size problem

To detect a 5-percentage-point difference in open rate at 95% confidence, you need approximately 500 subscribers per variant. To detect a 2-point difference, you need 3,000 per variant. Most smaller stores don't have the list size to run meaningful open rate tests, and that's fine, as long as you know it.

For lists under 2,000: don't run subject line A/B tests. Instead, manually try different approaches across sequential sends and look for patterns over time. For lists over 6,000: A/B testing becomes genuinely informative.

What to test (in priority order)

• Presence of a specific number: 'Your order is ready' vs. '1 item waiting for you'
• Length: under 35 characters vs. over 50 characters
• Question vs. statement: 'Ready to save 15%?' vs. 'Your 15% discount is ready'
• Personalisation: first name in subject line vs. none
• Urgency framing: 'Expires tonight' vs. 'Sale now on'

The right test structure

Don't do 50/50 splits. Use 40/40/20: 40% gets Variant A, 40% gets Variant B, 20% is held back. After 4 hours (when most open activity occurs), send the winner to the remaining 20%. This gives you a valid test while maximising sends to the winning variant.

Test one variable at a time. If you change the subject line length AND add a number AND change the tone, you don't know which change drove the result.

What Visisto's testing shows at scale

Across 180,000 A/B tested broadcasts in Q1 2026, the highest-impact single variable was presence of a specific number in the subject line, associated with a 4.2-percentage-point average open rate lift. Second: question format (+2.8pp). Third: first name personalisation (+1.9pp).

If you want A/B testing built into every broadcast with automatic winner selection, Visisto is free to start, no credit card: visisto.com