Your SlideShare is downloading. ×

A Culture of Testing


Published on

In the past few years, organizations like the Obama campaign and Upworthy have demonstrated how data and analytics can be used to run hugely successful digital programs. Many of the same principles …

In the past few years, organizations like the Obama campaign and Upworthy have demonstrated how data and analytics can be used to run hugely successful digital programs. Many of the same principles used by these groups can also be applied at smaller organizations to significantly improve the impact of online campaigns.
impact of online campaigns. This training will describe the “culture of testing” that’s needed at your own organization to run a data-driven program that maximizes engagement from your supporters.

Presenter: Jiim Pugh, Share Progress


Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 3. • Not just a set of rules or best practices • A commitment to: – Measure the effectiveness of your choices – Make decisions using data, not intuition • So… how do we do that? WHAT IS A CULTURE OF TESTING?
  • 4. • Metric: numerical measure of performance • Email metrics: – Open rate – Click rate – Action rate – Unsubscribe rate • Web page metrics: – Conversion rate (online actions per visitor) METRICS, METRICS, METRICS
  • 5. • Q: You have two ideas for an email – which one do you choose? • A: Both! • Steps to run an email A/B test: – Select two randomized email groups from your list – Send email A to the first group and email B to the second – Look at email metrics to determine which email did better – Send better email to the remainder of your list METRICS IN ACTION: EMAIL A/B TESTS
  • 6. • Example test: – Email A subject line about puppies, email B subject line about cats – List of 100,000 emails; select 10,000 randomly for A, 10,000 randomly for B – Send emails, measure results: • 1,276 open A and 1,403 open B • 267 donate to A, 195 donate to B – Which do you choose? METRICS IN ACTION: EMAIL A/B TESTS
  • 7. REAL EMAIL A/B TEST EXAMPLE 71% more in contributions from Version 1
  • 8. • Higher test performance doesn’t always mean better – needs to be statistically significant • Make sure test groups large enough for significance • CALCULATOR DEMO STATISTICAL SIGNIFICANCE
  • 9. • Emails aren’t free – lose people who unsubscribe • Need to balance benefit of actions against cost of unsubscribes • “Trial balloon” test: – Send email to small, randomized group – Evaluate metrics and decide if benefits outweigh costs – If yes, send to remainder; if no, don’t send TO SEND OR NOT TO SEND: UNSUBSCRIBES & TRIAL BALLOONS
  • 10. • Like email, possible to A/B test web pages • Web page A/B test: – Create several variations of web page – For new users coming to page, randomly show one variation – Record if visitor takes action – Calculate conversion rate for each variation – highest conversation rate is best • Need enough time for statistical significance WEB PAGE A/B TESTING
  • 11. 29% increased conversion rate WEB PAGE A/B TESTING
  • 12. • Until recently, not possible to do A/B testing on social media sharing • With ShareProgress, can now test Facebook and Twitter share posts: – Create several variations of Facebook post (title, description, thumbnail) or Twitter text – For users sharing from your website, randomly give them one variation – Record if users’ friends click share link and take action – Calculate recruitment rate for each variation – share post variation most likely to drive action is best SOCIAL A/B TESTING
  • 13. SOCIAL A/B TESTING Facebook Title Shares Driving Action / Total Shares Success Rate Improvement Monsanto shouldn’t be above the law 341/1041 32.8% (±2.9%) -- “One of the most outrageous special interest provisions in years” 322/962 33.5% (±3.0%) 2.2% Tell the Senate: Repeal the Monsanto Protection Act 458/972 47.1% (±3.1%) 43.8% Sign the Petition: Repeal the Monsanto Protection Act 555/1037 53.5% (±3.0%) 63.4% Facebook headline test by CREDO Action – 63% increased recruitment rate from Version 4
  • 14. • Emails and web pages • Social sharing • Online ads • Even offline actions CULTURE OF TESTING IN EVERYTHING YOU DO
  • 15. MAINTAINING A CULTURE OF TESTING • Get buy-in across the organization • Run at least one test for every campaign you do • Schedule periodic check-ins on what you've learned and accomplished
  • 16. • Email A/B testing: – Check your email management system • Webpage A/B testing: – Google Website Optimizer – Optimizely • Social A/B testing: – ShareProgress • A/B Test Calculator HELPFUL TOOLS
  • 17. QUESTIONS?