#emflconf@emfluence
Lindsay Waters, emfluence
lwaters@emfluence.com
Session Title A? Or Session Title B?
A/B Testing with Concorde Career Colleges
@lindsaymwaters
#emflconf@emfluence
Let’s give this a try…
Why A/B Test
Keys to A/B Testing
What to A/B test
How to A/B test
How to read results
Examples
#emflconf@emfluence
Be honest…
#emflconf@emfluence
#emflconf@emfluence
Keys to A/B Testing
Scientific
- Randomly select lists
- Test 1 element
- Random segmentation tool
Significant
- Results need to be
actionable
- Size of test segments
#emflconf@emfluence
Well what can I test?
• Tone
• Images
• Personalization
• Email Format
o Images
o Text
o Bullets
And then some…
• Time of Day
• Day of Week
• Subject Line/Pre-Header
o Length
o Offers
o Newsletter/Article Titles
• From Name
#emflconf@emfluence
How to A/B Test in the emfluence Platform
Step 1.
#emflconf@emfluence
How to A/B Test in the emfluence Platform
Step 2.
#emflconf@emfluence
How to A/B Test in the emfluence Platform
Step 3.
#emflconf@emfluence
How do I read my data?
Compare Unique Clicks:
- Personalization
- Email Format
- CTA Verbiage
- Copy
- Links
Compare Unique Opens:
- Day of Week
- Time of Day
- From Name
- Pre-header/Subject Lines
- Personalization
#emflconf@emfluence
#emflconf@emfluence
Happy Holidays – Alumni
Subject Line: Happy Holidays
from Concorde
Sent: Week of 12/19 @ 10 AM
Segment Size: 6,929
Testing: Day of Week
#emflconf@emfluence
Happy Holidays – Alumni Results
Day Sent
Unique
Views
Unique
Views %
Total
Views
Total
Views %
Unique
Clicks
Unique
Clicks %
Total
Clicks
Total
Clicks %
CTV
Web
Views
Web
Views %
Monday 1,220 17.60% 1,698 24.50% 29 0.42% 30 0.43% 2.40% 6 0.09%
Tuesday 1,251 18.00% 1,752 25.30% 33 0.48% 35 0.50% 2.60% 2 0.03%
Wednesday 1,250 18.00% 1,719 24.80% 30 0.43% 30 0.43% 2.40% 1 0.01%
Thursday 1,190 17.20% 1,667 24.10% 39 0.56% 39 0.56% 3.30% 3 0.04%
Friday 1,163 16.80% 1,640 23.60% 33 0.48% 34 0.49% 2.80% 1 0.01%
Totals 6,074 17.50% 8,476 24.50% 164 0.47% 168 0.48% 2.70% 13 0.04%
#emflconf@emfluence
Happy Holidays –
Prospective Students
Subject Line: Give yourself the gift
of education this holiday season
Sent: Thursday, Dec 22
Segment Size: 14,202
Testing: Time of Day
#emflconf@emfluence
Happy Holidays – Prospective Students Results
Time Sent
Unique
Views
Unique
Views %
Total
Views
Total
Views %
Unique
Clicks
Unique
Clicks %
Total
Clicks
Total
Clicks %
CTV
Web
Views
Web
Views %
Morning 1,790 12.60% 2,720 19.10% 130 0.91% 136 0.96% 7.30% 5 0.04%
Noon 2,033 14.30% 3,198 22.50% 159 1.10% 177 1.20% 7.80% 9 0.06%
Afternoon 1,909 13.40% 3,070 21.60% 124 0.87% 132 0.93% 6.50% 4 0.03%
Evening 2,179 15.30% 3,399 23.90% 142 1.00% 147 1.00% 6.50% 7 0.05%
Totals 7,911 13.90% 12,387 21.80% 555 0.98% 592 1.00% 7.00% 25 0.04%
#emflconf@emfluence
Alumni Newsletter – Q1
Subject Line: Introducing your
newly named alumni newsletter
Sent: Week of 1/11 @ 3:30 PM
Segment Size: 4,973
Testing: Day of Week (including
weekends)
#emflconf@emfluence
Alumni Newsletter – Q1 Results
ResultsDay Sent
Unique
Views
Unique
Views %
Total
Views
Total
Views %
Unique
Clicks
Unique
Clicks %
Total
Clicks
Total
Clicks %
CTV
Web
Views
Web
Views %
Wednesday 1,098 22.10% 1,673 33.60% 51 1.00% 52 1.00% 4.60% 4 0.08%
Thursday 1,061 21.30% 1,514 30.40% 44 0.88% 46 0.92% 4.10% 1 0.02%
Friday 1,045 21.00% 1,598 32.10% 46 0.92% 63 1.30% 4.40% 0 0.00%
Saturday 1,116 22.40% 1,868 37.60% 48 0.97% 58 1.20% 4.30% 4 0.08%
Sunday 1,054 21.20% 1,623 32.60% 41 0.82% 53 1.10% 3.90% 2 0.04%
Monday 1,102 22.20% 1,706 34.30% 38 0.76% 39 0.78% 3.40% 2 0.04%
Tuesday 1,107 22.30% 1,620 32.60% 55 1.10% 64 1.30% 5.00% 4 0.08%
Totals 7,583 21.80% 11,602 33.30% 323 0.93% 375 1.10% 4.30% 17 0.05%
#emflconf@emfluence
Alumni Newsletter – Q2
Subject Line: Spring is here, and
so is your Insight newsletter
Sent: Thursday, April 20 @ 6 PM
Segment Size: 18,092
Testing: Email Layout
#emflconf@emfluence
Alumni Newsletter – Q2 Results
Email
Creative
Unique
Views
Unique
Views %
Total
Views
Total
Views %
Unique
Clicks
Unique
Clicks %
Total
Clicks
Total
Clicks %
CTV
Web
Views
Web
Views %
A - Bullets 3,486 19.30% 5,311 29.40% 206 1.10% 235 1.30% 5.90% 2 0.01%
B - Paragraph 3,360 18.60% 4,961 27.40% 203 1.10% 218 1.20% 6.00% 3 0.02%
Totals 6,846 18.90% 10,272 28.40% 409 1.10% 453 1.30% 6.00% 5 0.01%
Creative
A
Creative
B
#emflconf@emfluence
Alumni Newsletter - Q3
Subject Line: Gain “Insight”
with latest Concorde
Alumni Newsletter
Sent: Friday, July 28 @
3:30 PM
Segment Size: 18,948
Testing: Email Layout
#emflconf@emfluence
Alumni Newsletter – Q3 Results
Email
Creative
Unique
Views
Unique
Views %
Total
Views
Total
Views %
Unique
Clicks
Unique
Clicks %
Total
Clicks
Total
Clicks %
CTV
Web
Views
Web
Views %
A - Image 2,836 15.00% 3,761 19.80% 195 1.00% 231 1.20% 6.90% 7 0.04%
B - Text 2,787 14.70% 3,758 19.80% 172 0.91% 212 1.10% 6.20% 1 0.01%
Totals 5,623 14.80% 7,519 19.80% 367 0.97% 443 1.20% 6.50% 8 0.02%
#emflconf@emfluence
Email Address
Subject Line: We’re here to help you
Sent: Thursday, May 18 @ 2 PM
Segment Size: 6,710
Testing: Personal vs. Concorde
Email Address
#emflconf@emfluence
Personal vs. Concorde Email
Email
Creative
Unique
Views
Unique
Views %
Total
Views
Total
Views %
Unique
Clicks
Unique
Clicks %
Total
Clicks
Total
Clicks %
CTV
Web
Views
Web
Views %
Personal 2,476 38.70% 4,259 66.50% 22 0.34% 22 0.34% 0.89% 8 0.12%
Concorde 1,488 21.20% 3,160 45.10% 21 0.30% 24 0.34% 1.40% 10 0.14%
Totals 3,964 29.50% 7,419 55.30% 43 0.32% 46 0.34% 1.10% 18 0.13%
#emflconf@emfluence

Session Title A or Session Title B: A/B Testing with Concorde Career Colleges

  • 1.
    #emflconf@emfluence Lindsay Waters, emfluence lwaters@emfluence.com SessionTitle A? Or Session Title B? A/B Testing with Concorde Career Colleges @lindsaymwaters
  • 2.
    #emflconf@emfluence Let’s give thisa try… Why A/B Test Keys to A/B Testing What to A/B test How to A/B test How to read results Examples
  • 3.
  • 4.
  • 5.
    #emflconf@emfluence Keys to A/BTesting Scientific - Randomly select lists - Test 1 element - Random segmentation tool Significant - Results need to be actionable - Size of test segments
  • 6.
    #emflconf@emfluence Well what canI test? • Tone • Images • Personalization • Email Format o Images o Text o Bullets And then some… • Time of Day • Day of Week • Subject Line/Pre-Header o Length o Offers o Newsletter/Article Titles • From Name
  • 7.
    #emflconf@emfluence How to A/BTest in the emfluence Platform Step 1.
  • 8.
    #emflconf@emfluence How to A/BTest in the emfluence Platform Step 2.
  • 9.
    #emflconf@emfluence How to A/BTest in the emfluence Platform Step 3.
  • 10.
    #emflconf@emfluence How do Iread my data? Compare Unique Clicks: - Personalization - Email Format - CTA Verbiage - Copy - Links Compare Unique Opens: - Day of Week - Time of Day - From Name - Pre-header/Subject Lines - Personalization
  • 11.
  • 12.
    #emflconf@emfluence Happy Holidays –Alumni Subject Line: Happy Holidays from Concorde Sent: Week of 12/19 @ 10 AM Segment Size: 6,929 Testing: Day of Week
  • 13.
    #emflconf@emfluence Happy Holidays –Alumni Results Day Sent Unique Views Unique Views % Total Views Total Views % Unique Clicks Unique Clicks % Total Clicks Total Clicks % CTV Web Views Web Views % Monday 1,220 17.60% 1,698 24.50% 29 0.42% 30 0.43% 2.40% 6 0.09% Tuesday 1,251 18.00% 1,752 25.30% 33 0.48% 35 0.50% 2.60% 2 0.03% Wednesday 1,250 18.00% 1,719 24.80% 30 0.43% 30 0.43% 2.40% 1 0.01% Thursday 1,190 17.20% 1,667 24.10% 39 0.56% 39 0.56% 3.30% 3 0.04% Friday 1,163 16.80% 1,640 23.60% 33 0.48% 34 0.49% 2.80% 1 0.01% Totals 6,074 17.50% 8,476 24.50% 164 0.47% 168 0.48% 2.70% 13 0.04%
  • 14.
    #emflconf@emfluence Happy Holidays – ProspectiveStudents Subject Line: Give yourself the gift of education this holiday season Sent: Thursday, Dec 22 Segment Size: 14,202 Testing: Time of Day
  • 15.
    #emflconf@emfluence Happy Holidays –Prospective Students Results Time Sent Unique Views Unique Views % Total Views Total Views % Unique Clicks Unique Clicks % Total Clicks Total Clicks % CTV Web Views Web Views % Morning 1,790 12.60% 2,720 19.10% 130 0.91% 136 0.96% 7.30% 5 0.04% Noon 2,033 14.30% 3,198 22.50% 159 1.10% 177 1.20% 7.80% 9 0.06% Afternoon 1,909 13.40% 3,070 21.60% 124 0.87% 132 0.93% 6.50% 4 0.03% Evening 2,179 15.30% 3,399 23.90% 142 1.00% 147 1.00% 6.50% 7 0.05% Totals 7,911 13.90% 12,387 21.80% 555 0.98% 592 1.00% 7.00% 25 0.04%
  • 16.
    #emflconf@emfluence Alumni Newsletter –Q1 Subject Line: Introducing your newly named alumni newsletter Sent: Week of 1/11 @ 3:30 PM Segment Size: 4,973 Testing: Day of Week (including weekends)
  • 17.
    #emflconf@emfluence Alumni Newsletter –Q1 Results ResultsDay Sent Unique Views Unique Views % Total Views Total Views % Unique Clicks Unique Clicks % Total Clicks Total Clicks % CTV Web Views Web Views % Wednesday 1,098 22.10% 1,673 33.60% 51 1.00% 52 1.00% 4.60% 4 0.08% Thursday 1,061 21.30% 1,514 30.40% 44 0.88% 46 0.92% 4.10% 1 0.02% Friday 1,045 21.00% 1,598 32.10% 46 0.92% 63 1.30% 4.40% 0 0.00% Saturday 1,116 22.40% 1,868 37.60% 48 0.97% 58 1.20% 4.30% 4 0.08% Sunday 1,054 21.20% 1,623 32.60% 41 0.82% 53 1.10% 3.90% 2 0.04% Monday 1,102 22.20% 1,706 34.30% 38 0.76% 39 0.78% 3.40% 2 0.04% Tuesday 1,107 22.30% 1,620 32.60% 55 1.10% 64 1.30% 5.00% 4 0.08% Totals 7,583 21.80% 11,602 33.30% 323 0.93% 375 1.10% 4.30% 17 0.05%
  • 18.
    #emflconf@emfluence Alumni Newsletter –Q2 Subject Line: Spring is here, and so is your Insight newsletter Sent: Thursday, April 20 @ 6 PM Segment Size: 18,092 Testing: Email Layout
  • 19.
    #emflconf@emfluence Alumni Newsletter –Q2 Results Email Creative Unique Views Unique Views % Total Views Total Views % Unique Clicks Unique Clicks % Total Clicks Total Clicks % CTV Web Views Web Views % A - Bullets 3,486 19.30% 5,311 29.40% 206 1.10% 235 1.30% 5.90% 2 0.01% B - Paragraph 3,360 18.60% 4,961 27.40% 203 1.10% 218 1.20% 6.00% 3 0.02% Totals 6,846 18.90% 10,272 28.40% 409 1.10% 453 1.30% 6.00% 5 0.01% Creative A Creative B
  • 20.
    #emflconf@emfluence Alumni Newsletter -Q3 Subject Line: Gain “Insight” with latest Concorde Alumni Newsletter Sent: Friday, July 28 @ 3:30 PM Segment Size: 18,948 Testing: Email Layout
  • 21.
    #emflconf@emfluence Alumni Newsletter –Q3 Results Email Creative Unique Views Unique Views % Total Views Total Views % Unique Clicks Unique Clicks % Total Clicks Total Clicks % CTV Web Views Web Views % A - Image 2,836 15.00% 3,761 19.80% 195 1.00% 231 1.20% 6.90% 7 0.04% B - Text 2,787 14.70% 3,758 19.80% 172 0.91% 212 1.10% 6.20% 1 0.01% Totals 5,623 14.80% 7,519 19.80% 367 0.97% 443 1.20% 6.50% 8 0.02%
  • 22.
    #emflconf@emfluence Email Address Subject Line:We’re here to help you Sent: Thursday, May 18 @ 2 PM Segment Size: 6,710 Testing: Personal vs. Concorde Email Address
  • 23.
    #emflconf@emfluence Personal vs. ConcordeEmail Email Creative Unique Views Unique Views % Total Views Total Views % Unique Clicks Unique Clicks % Total Clicks Total Clicks % CTV Web Views Web Views % Personal 2,476 38.70% 4,259 66.50% 22 0.34% 22 0.34% 0.89% 8 0.12% Concorde 1,488 21.20% 3,160 45.10% 21 0.30% 24 0.34% 1.40% 10 0.14% Totals 3,964 29.50% 7,419 55.30% 43 0.32% 46 0.34% 1.10% 18 0.13%
  • 24.