Metrics and Testing

1,069 views
1,029 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,069
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Metrics and Testing

  1. 1. Stats are Sexy!Metrics and Testing That’s HOT! 1
  2. 2. 2 “The only truth is what actually happens.”The number of people who did thething you wanted them to do. Give. Act. Volunteer. Come to Get Together. 2
  3. 3. 3The data Oh NO! Let’s take a closer look... 3
  4. 4. 4 What are all these metrics?• It’s easy to get lost in the result metrics, some common calculations are: • Delivery rate • Open rate • Click-through rate • Conversion rate • Tell-a-Friend rates 4
  5. 5. 5Delivery Rate = Deliveryemails Received/ Number of Total emails Sent Delivered Emails, not yet opened 5
  6. 6. 6 Open Rate = Opening the email members Number of emailed who opened at least 1 email/Opened emailTotal Emails Sent 6
  7. 7. 7Beware, Preview Pane Affects Open Rates 7
  8. 8. 8Opened email 8
  9. 9. 9Click-Through Click-through Rate = Number of emailed members who clicked on at least one link/ Total Emails Sent 9
  10. 10. 10The Response Response Rate = Number of emailed members who participated in the campaign / Total emails Sent 10
  11. 11. 11Tell-a-Friend Usage Tell-a-Friend Usage Rate = Number of emailed members who told at least 1 friend/ Total emails Sent 11
  12. 12. 12Tell-a-friend Recruitment Tell-a-Friend Recruitment Rate = Number of Friends who joined a center/ Total Emails Sent 12
  13. 13. 13 Which Rates are the most important?• Conversion Rates are the best measure of a campaign’s immediate success• Click-through rates can be good indicators of how members are interacting with your messages• Open rates are deceiving for short term data, but if measured over several years can become interesting. They also are good for normalizing other data. 13
  14. 14. 14So, what can you do with all these statistics? 14
  15. 15. 15 This is a numbers game … … do the math• If you want 10 people to TAKE ACTION, then: • (at least) 20 people need to click to the advocacy page • (at least) 30 people need to open your e-mail message • (at least) 100 people need to receive your message 15
  16. 16. 16 This is a numbers game … math … do the• If you want 10 people to DONATE, then: • (at least) 50 people need to click to the donation page • (at least) 250 people need to open your e- mail message • (at least) 1,000 people need to receive your message• Yes, your results may vary…dramatically! 16
  17. 17. 17 It all starts with the list• Good direct marketing is all about the list • It starts with the list • It ends with the list • It is all about the list• But building a good list requires you to be good at everything else: your work, copy, layout, timing, etc. 17
  18. 18. 18 Typical response rates• Nothing is typical, really• Big lists usually = lower response…• …and smaller lists usually ROCK• But if you make us… 18
  19. 19. 19 Where can I find my industry standards?• Marketingsherpa.com• Emarketer.com• Opt-In News• Your own campaigns! 19
  20. 20. 20Making the most of the numbers to get the most out of your online program… Segmenting and Testing 20
  21. 21. 21 Segmenting is……dividing your e-mail list when running campaigns tolearn more, personalize, and generate higher results. 21
  22. 22. 22 Segmentation example• Donors Recipients Amt of # of Avg Resp. Gifts Gifts gift rate 3,500 $2,290 40 $57.75 1.13%• People on direct mail list with e-mail Recipients Amt of # of Avg Resp. Gifts Gifts gift rate 7,250 $3,385 92 $36.79 1.26%• E-mail subscribers Recipients Amt of # of Avg Resp. Gifts Gifts gift rate 7,000 $775 26 $29.81 0.37% 22
  23. 23. 23Before segmenting… 23
  24. 24. 24After segmenting… 24
  25. 25. 25 Other segmentation strategies• Giving level• Campaign-based • Donors who have taken action • Donors who have not taken action • Non-donors who have taken action • Non-donor, non-action-takers• Recency• Source• One-time and lapsed online donors• Active and inactive non-donors 25
  26. 26. 26 Testing ……Once you’ve determined how best to segment yourparticular list, you can begin to test what works withineach of these segments. 26
  27. 27. 27 Testing• Open rates • Time of day, day of week • Subject lines • Straightforward subject lines vs suggestive ones. • Your action will help a child eat today vs. • How can you help the orphans? • To Mention The Ask Or Not? • Looking ahead, more animals need your help in 2007 vs. • Last chance for your tax-deductible gift to animals • Subject lines that in different ways ask for a renewal of commitment. • Renew your commitment for 2007 vs. • Your membership is expiring vs. • Renew today—and save even more puppies 27
  28. 28. 28 Testing, continued• Click through rates • Premiums • Offer • Signer • Short or long copy • Link placement• Conversion rates • Short vs. long page • Images and insets • First things first! • Giving levels 28
  29. 29. 29 How to create a good test...• Pick one variable you want to test• Keep everything else the same• Randomly 50/50 split the population of the campaign• Compare the results from each test group using a “significance calculator”• “Significant” results mean the results can be repeated 29
  30. 30. 30 Testing Do’s and Don’t’s• DO • Ask a question and design a test to answer it • Select a RANDOM sample of your list • Test ONE thing at a time • Keep everything the same except for the thing you are testing • Check for statistical significance • Make sure your list segments are large enough to produce meaningful results • Understand that some test results are only applicable to the message you were testing 30
  31. 31. 31 Testing Do’s and Don’t’s• DON’T • Assume that test results apply across lists or list segments • Bite off more than you can chew • Read too much into the numbers • Assume that every test will yield statistically significant results 31
  32. 32. 32 Testing Gone Wrong Click Response Better Statistical Messages Through Unique Rate (unique Than SignificanceVersion Description Sent Open Rate Rate Signatures sigs) Control? ?CONTROL Regular Email 21,059 20.90% 8.40% 1,395 6.62% Masked URLs – used hyperlinked text insteadTest 1 of spelled out URL 20,953 22.70% 10.60% 1,645 7.85% Y Y In Line URLs –URLs are in the line of text, notTest 2 separate paragraphs 20,377 21.30% 8.30% 1,318 6.47% N N Longer subject line – Everything is same as control, except the subjectTest 3 line is a longer version 20,419 20.30% 7.90% 1,262 6.18% N N Header Images- Everything is same as control, except header image does not have aTest 4 picture of candidate 20,784 21.50% 8.30% 1,391 6.69% Y N No Header Image –Test 5 header image is removed 21,734 22.50% 8.80% 1,524 7.01% Y N Large Button - take action button in the upperTest 6 right hand corner is larger 21,014 21.10% 8.50% 1,431 6.81% Y N Move On Style- email is stripped of header imageTest 7 and all graphics 21,591 22.60% 10.70% 1,804 8.36% Y Y Short Email – middleTest 8 paragraphs are removed 21,146 23.90% 8.90% 1,483 7.01% Y N Greeting - lead in to the email says “Dear Friend” 32Test 9 instead of Dear First Name 20,702 21.30% 7.90% 1,328 6.41% N NTotal 209,779 21.82% 8.84% 14,581 6.95% Y N
  33. 33. 33 What Happened?• Too many variables• Tried to answer every question at once• The result: • Hours of work to set up 10 segments • Data was messy and difficult to decipher • We didn’t learn much about our list 33
  34. 34. 34 What We Should Have Done• Picked one variable• Had a clear question to answer• Split the list in half or thirds instead of in tenths 34
  35. 35. 35 And the winner is….• Subject Line Test • What metric should be used to evaluate a subject line? • Which is the winner? 35
  36. 36. 36 And the winner is…• Open Rate• Two-way tie Click Open Through Response Message Emails Sent Rate Rate Actions Rate Bushs Budget: Guns Not Butter 1,000 28.50% 13.00% 53 5.30% Whats At Stake 999 31.90% 15.50% 52 5.20% Action Alert: Defend 100 Hours Agenda 999 27.70% 14.80% 49 4.90% Action Alert: Organize agaist Bushs budget 1,000 31.90% 18.20% 51 5.10% Organize Against Bushs Budget 1,000 28.80% 16.70% 54 5.40% Total 4,998 30.60% 16.90% 259 5.18% 36
  37. 37. 37 And the winner is…• Link placement test • Which is the most important metric? • Which segment is the winner? Click Messages Through Unique ResponseTest Sent Open Rate Rate Signatures RateBelow the firstparagraph 20,419 22.70% 10.60% 1,645 8.06%Below the thirdparagraph 20,567 21.30% 8.30% 1,318 6.41%Below the fifthparagraph 20,489 20.30% 7.90% 1,262 6.18% 37
  38. 38. 38 And the winner is…• Click Through Rate• Below the first paragraph 38
  39. 39. 39 And the winner is….• Short Email vs. Long Email • What metric(s) should you look at? • Which message is the winner? Click Messages Through Response Sent Open Rate Rate RateLong Email 95,917 20.90% 7.50% 5.62%Short Email 96,953 21.10% 7.30% 5.48% 39
  40. 40. 40 And the winner is… • Click through rates and response rates • Trick question – the results are statistically insignificant. • You can’t always eyeball test results – be sure to test for statistical significance Click Messages Through Response Sent Open Rate Rate RateLong Email 95,917 20.90% 7.50% 5.62%Short Email 96,953 21.10% 7.30% 5.48% 40
  41. 41. 41 And the winner is… • Which campaign is more successful? Why? Click Messages Open Through Amount Average ResponseCampaign Received Rate Rate Donations Donated Gift RateIssue #1 63,101 42.06% 0.49% 102 $6,578 $64.49 0.16%Issue #2 62,653 41.48% 0.62% 125 $4,960 $39.68 0.20% 41
  42. 42. 42 And the winner is…• Arctic campaign most likely to raise money (fewer donors, but higher average gift)• Public lands campaign most likely to get engage donors (more donors, lower average gift). • Depending on the goals of the campaign (raise a lot of $, or get more people to give), both are a success. 42
  43. 43. 43 And the winner is….• From line testing – by list segment• What should the organization do moving forward? 43
  44. 44. 44 And the winner is…• Even thought the inactive segment had a lower open rate with the from line “Kathy, Organization” all of the other segments saw improved open rates.• You could segment out the inactive folks and continue to send them messages with the old from line, but it’s probably not worth the effort. 44
  45. 45. 45 Digging Deeper:5 Questions you could ask 45
  46. 46. 46 Question 1Which group of people on myemail list is the most likely totake action, tell a friend, ordonate? 46
  47. 47. 47Question 1, Sample answer Source 1 Source 2Members in Group 2,403 4,014Activists* 59.84% 29.45%Donors 5.45% 2.72%TAF 35.41% 9.12%Super activists 4.99% 2.99%Members who opened an email 57.26% 55.23%in 2005Members who clicked 70.41% 54.53% 47
  48. 48. 48 Answering Question 1• Two words: Source Codes• Suffix on imports or on any link someone may click to come on your list, starts with ‘? source=’ • Advocacy campaign (?source=advo1) • Viral Flash game or movie (?source=funny_movie) • Banner ads (?source=yahoo_banner) • Paid source (?source=care2) • Donation campaign (?source=moneybags)• Source codes are attached to a member so you always know where they originally came 48 from
  49. 49. 49 Answering Question 1• Each cell in the table is a member select• Query members who have the source code you want AND have donated, taken action, etc.• The results can help you compare how valuable names from various sources are to your organization 49
  50. 50. 50 Question 2How do I know how fast mylist is growing? 50
  51. 51. 51 Answering Question 2• List Growth Rate: # new sign-ups/# of deliverable addresses • Shows % list growth over a period of time (month, 3-months, year)• List Hurdle Rate: # removals/# of deliverable addresses • Shows % of list that leaves over a period of time • # removals tells you how many new names you need to get before you achieve net growth 51
  52. 52. 52 Question 3How come my advocacyresponse rates are so low? 52
  53. 53. 53 Question 4How do I find out if I’msending too much email? 53
  54. 54. 54 Question 5How valuable is an emailaddress? 54

×