A/B Testing Popups: 14 Experiments, 1 Clear Winner

Visisto is the complete website growth system with built-in A/B testing for all 20 widget types. We ran 14 controlled experiments across spin wheels, slide-ins, popups, and sticky bars to find the exact combination that maximises email capture rate. The winning setup increased capture rate by 340% over the control. Here's every test, every result, and the final formula.

How we tested: methodology

Every test followed the same protocol to ensure clean results:

• Duration: 14 days minimum per test
• Traffic: 1,000+ unique impressions per variant
• Split: 50/50 random traffic allocation
• Significance: 95% confidence threshold before declaring a winner
• One variable: Each test changed exactly one element
• Platform: Visisto's built-in A/B testing, no external tools

Tests 1, 3: Widget type (the biggest lever)

We started with the variable that has the largest impact: what kind of widget visitors see.

Test 1: Standard popup vs spin wheel

• Control: Standard discount popup, "Get 10% off. Enter your email." → 2.1% capture rate
• Variant: Spin wheel with same 10% prize + other prizes → 7.8% capture rate
• Winner: Spin wheel (+271%). Confirmed at 99.8% confidence.

Test 2: Spin wheel vs scratch card

• Control: Spin wheel (6 prizes) → 8.1% capture rate
• Variant: Scratch card (same prizes hidden) → 5.4% capture rate
• Winner: Spin wheel (+50%). The visible prize options create more anticipation than hidden rewards.

Test 3: Popup vs slide-in

• Control: Centre popup with overlay → 2.8% capture rate
• Variant: Bottom-right slide-in → 3.4% capture rate
• Winner: Slide-in (+21%). Less intrusive = less dismissal. But both lose to spin wheels.

Tests 4, 6: Trigger timing

Test 4: Page load vs exit intent

• Control: Show immediately on page load → 1.9% capture rate
• Variant: Show on exit intent → 4.2% capture rate
• Winner: Exit intent (+121%). Page load popups also increased bounce rate by 11%.

Test 5: Time delay, 5s vs 15s vs 30s

• 5 seconds: 2.2% capture rate, too aggressive, visitors haven't engaged yet
• 15 seconds: 3.1% capture rate, sweet spot
• 30 seconds: 2.8% capture rate, many visitors leave before seeing the widget
• Winner: 15-second delay. Long enough to build context, short enough to catch most visitors.

Test 6: Scroll depth, 30% vs 50% vs 70%

• 30% scroll: 2.6% capture rate
• 50% scroll: 3.6% capture rate
• 70% scroll: 3.3% capture rate
• Winner: 50% scroll depth. 70% misses visitors who engage but don't scroll all the way.

Tests 7, 10: Offer and copy

Test 7: Percentage discount vs free shipping

• Control: "Get 10% off" → 3.2% capture rate
• Variant: "Get free shipping" → 3.7% capture rate
• Winner: Free shipping (+16%). Free shipping feels like removing a penalty; discounts feel like a deal. Loss aversion wins.

Test 8: Specific vs vague CTA copy

• Control: "Subscribe for exclusive deals" → 1.4% capture rate
• Variant: "Get your 10% code now" → 3.1% capture rate
• Winner: Specific (+121%). Always tell visitors exactly what they get.

Test 9: With countdown timer vs without

• Control: Spin wheel, no timer → 7.6% capture rate
• Variant: Spin wheel + 15-min timer after spin → 9.3% capture rate
• Winner: Timer (+22%). The timer activates loss aversion after the visitor has already "won" a prize.

Test 10: Email field before vs after spin

• Control: Enter email → then spin → 5.2% capture rate
• Variant: Spin → see prize → enter email to claim → 7.3% capture rate
• Winner: Email after spin (+40%). Let visitors play first. The prize creates reciprocity.

Tests 11, 14: Design and UX

Test 11: Brand colours vs high-contrast

• Control: On-brand muted palette → 6.8% capture rate
• Variant: High-contrast neon/bright palette → 8.0% capture rate
• Winner: High contrast (+18%). Widgets should feel like games, not branded collateral.

Test 12: Number of spin wheel slots (4 vs 6 vs 8)

• 4 slots: 6.9% capture rate, feels limited
• 6 slots: 8.2% capture rate, enough variety, not overwhelming
• 8 slots: 7.4% capture rate, too much choice, prize text shrinks
• Winner: 6 slots (+12% over 4-slot).

Test 13: Prize reveal, instant vs suspense animation

• Control: Instant stop → 7.8% capture rate
• Variant: 3-second suspense animation → 8.4% capture rate
• Winner: Suspense (+8%). The delay builds emotional investment in the outcome.

Test 14: Mobile, partial overlay vs full-screen

• Control: Partial overlay (desktop-style) → 4.2% mobile capture rate
• Variant: Full-screen takeover → 7.1% mobile capture rate
• Winner: Full-screen (+69%). Mobile needs its own design, not a scaled-down desktop popup.

The winning formula: 340% more captures

Combining every winning variant from all 14 tests produces the optimal widget configuration:

• Widget type: Spin wheel (6 slots, high-contrast colours)
• Trigger: Exit intent OR (50% scroll + 15s delay)
• Email timing: After spin (see prize first, then enter email)
• Post-claim: 15-minute countdown timer for prize redemption
• Prize reveal: 3-second suspense animation
• Mobile: Full-screen takeover
• Frequency: Once per visitor per 7 days

This configuration captures 8.3% of visitors, vs 1.9% for a page-load standard popup (the most common setup on Shopify stores). That's a 340% improvement.

IMPACT ON A STORE WITH 10,000 MONTHLY VISITORS
────────────────────────────────────────────────
Before (standard popup):  190 captures/month
After (optimised wheel):  830 captures/month
Difference:              +640 captures/month
Annual extra contacts:   +7,680

At £15 avg subscriber value = £115,200 annual revenue impact

The bottom line

A/B testing isn't optional, it's the difference between a 1.9% and 8.3% capture rate. But you don't need 14 tests. Start with the biggest lever (widget type), then optimise trigger timing and offer. The full formula above took us 6 months of testing. You can implement it in 15 minutes on Visisto.

Visisto includes built-in A/B testing on every plan, free included. No separate testing tool needed. Set up your first experiment today and let the data decide.

A/B testing email subject lines: the variables that actually matter

Email subject line A/B tests are the most commonly run test in email marketing and also the most commonly misinterpreted. Open rate is not revenue. A subject line that generates a 40% open rate but attracts curiosity-clicks from non-buyers will underperform a 28% open rate subject line that attracts genuine purchase intent. Best practice is to measure A/B test outcomes at the level of conversion, click-to-purchase rate or revenue per send, rather than open rate alone. The variables with the highest impact on subject line performance are: personalisation tokens (first name in subject lifts opens 2–5% on average), curiosity gaps, number-led subjects, and urgency signals.

Statistical significance in email A/B tests: when can you trust a result?

Most marketing teams declare A/B test winners too early. A test showing 24% opens for variant A vs 18% opens for variant B after 200 sends per variant has insufficient data to be statistically significant. The minimum sample size for a reliable email A/B test is 1,000 sends per variant for open rate comparisons, and 2,500–5,000 sends per variant if you are measuring conversion rate. Visisto calculates statistical significance automatically and will not declare a winner until the 95% confidence threshold is reached.

A/B testing across the full funnel: widgets, emails, and sequences

The highest-leverage A/B tests are at the widget level and the sequence level. A widget A/B test run on exit intent affects every contact you capture going forward. A sequence A/B test affects the lifetime value of every subscriber in that flow. Visisto runs A/B tests at all three levels simultaneously: widget variants split traffic at the point of capture, email variants test individual messages within sequences, and sequence variants test entire flow architectures against each other. Running all three in parallel compounds the improvement rate significantly faster than sequential single-variable testing.

How to interpret your A/B test results without fooling yourself

The most common mistake in A/B testing is peeking at results before the test reaches statistical significance and stopping when you see a winner. This practice — called optional stopping — produces false positive rates of 25–50% rather than the intended 5%. The correct process is to set a minimum sample size before starting the test, not look at the results until that threshold is reached, and then read the result once. Visisto enforces this automatically by hiding the winner indicator until the configured significance level is reached. You can monitor traffic allocation and total impressions, but the winner highlight only appears when the data supports it.

The most impactful A/B tests to run in 2026: ranked by expected lift

1. Popup format test (static modal vs. spin wheel): average 4–8pp capture rate improvement, highest single-test impact available
2. Trigger timing (exit intent vs. 8-second delay): average 1.5–3pp improvement, affects every impression
3. Email subject line test (curiosity vs. benefit-led): average 4–8pp open rate improvement
4. CTA button copy test (action word change): average 2–4pp click-through improvement
5. Offer type test (% discount vs. free shipping): varies by store, can shift capture rate 2–5pp
6. Send time test (morning vs. evening vs. weekend): average 3–6pp open rate variation by audience segment

How to run A/B tests that sales and finance will actually believe

The challenge with A/B test results in most organisations is credibility. Marketing reports 'variant B had 23% higher click-through rate' and finance asks 'did it actually generate more revenue?' The answer depends on whether you measured at the right level. For A/B tests on capture widgets, the metric that finance cares about is cost per acquired subscriber and 90-day revenue per subscriber. For email A/B tests, the metric is revenue per send, not open rate. For sequence A/B tests, the metric is average order value and repeat purchase rate from each variant's subscribers. Visisto tracks revenue attribution from first popup impression to completed purchase, giving you the full funnel numbers that make A/B results credible to non-marketing stakeholders.

What to do after your A/B test reaches significance

Most teams stop at significance declaration: variant B won, ship variant B. The next-level approach extracts more value from the same experiment. First, segment the result by traffic source — the winning variant may perform differently for paid vs. organic visitors, and the insight may justify separate widget configurations per source. Second, document the causal hypothesis: why did variant B win? A spin wheel outperformed a static modal not just because of gamification but likely because it created anticipation before the email field appeared — that insight should inform the next test. Third, size the annual value of the improvement: if the test lifted capture rate from 3.2% to 7.8% on 10,000 monthly visitors, the annualised gain is 5,520 additional subscribers per year. At a £20 LTV per subscriber, the single test was worth £110,400 in expected annual value.