Your SlideShare is downloading. ×
0
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
BreakItOff Evaluation
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

BreakItOff Evaluation

626

Published on

Presentation delivered at the Canadian Evaluation Society Conference in Toronto in June 2013. The results and process behind the development evaluation and outcome evaluation with the Break It Off …

Presentation delivered at the Canadian Evaluation Society Conference in Toronto in June 2013. The results and process behind the development evaluation and outcome evaluation with the Break It Off social media campaign for youth smoking cessation were discussed.

Published in: Health & Medicine, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
626
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Evaluation of Break It Off: a Social Media-based Campaign Targeting Young Adult Smokers Cameron D. Norman1 N.Bruce Baskerville2 ,Nancy Korstanje3 Steve Brown2, Barbara Zupko2 ,Larry Squire2 1 CENSE Research + Design & Dalla Lana School of Public Health, University of Toronto 2Propel Centre for Population Health Impact, 3The Canadian Cancer Society Canadian Evaluation Society Conference, Toronto, ON June 2013
  • 2. BACKGROUND
  • 3.   Young adults (19-29) have the highest rate of smoking in Canada – 24.4% use Tobacco (CCHS, 2010)   86% of young adults (18 to 34 years) in Canada are using social media tools – e.g Facebook, Twitter, etc. (Ipsos Reid, 2011)   Social media tools are immensely popular, yet not well studied in terms of population health impact   Evidence to-date suggest wide reach, low barriers to use, and portability → great potential for population level impact     3  
  • 4. Any electronic, networked information resource that derives its principal value from user contributions, engagement & interaction
  • 5. Developmental Evaluation
  • 6. Developmental Evaluation •  First proposed by Michael Quinn Patton (1994) •  Combines Utilization-Focused Evaluation with concepts from Complexity Science •  Social innovation and resiliency require actionable feedback to enable programs to adapt and thrive through change •  Strategic learning / real-time evaluation
  • 7. Developmental Evaluation •  Assists social innovators to learn strategically •  Supports adaptation to changing conditions •  Develops promising interventions •  Documents innovators’ actions •  Identifies emergent processes and outcomes; enhances resiliency •  Determines if an innovation is ready for formative evaluation
  • 8. Systems Thinking •  Knowledge structures can be built upon: –  Information: symbolic pattern of matter and energy –  Networks: the information delivery channels and attendant limits –  Systems: Bounded elements linked to facilitate the flow of information and support networks
  • 9. Break  It  Off   11     Promoted January 2012 to March 2012 •  Paid advertising on websites with high levels of young adult viewership. •  Banner ads on sites such as Microsoft, Yahoo and Facebook. •  Homepage takeovers on the Weather Network. •  Ads on dating sites such as Plenty of Fish and OK Cupid. •  Facebook, Twitter, YouTube videos and blogs.   Links to Smokers’ Helpline
  • 10. What are Quitlines?   Quitlines are computer-supported, telephone-based tobacco cessation counselling services that help tobacco users quit involving a live, trained cessation counsellor.   Quitlines reach and serve tobacco users, regardless of location.   Residents in all 50 states, the District of Columbia, each U.S. territory, all ten Canadian provinces and two territories have access to public Quitline services.   Smokers’ Helpline (SHL) is an evidence-based cessation service managed by the Canadian Cancer Society.   SHL reach into the young adult smoking population is estimated to be just 0.5%. 12  
  • 11. Developmental  Evalua4on  Research     Approach   13   Step  5:  Act   In  collabora.on  with  CCS,  synthesis   and  transla.on  of  knowledge  into   publica.on,  programming  and/or   future  evalua.on  research.     Step  1:  Listen   Interact  with  CCS  to  iden.fy  their   needs  and  wants  and  build  a  shared   understanding  of  the  context,   objec.ves,  challenges  and   opportuni.es  associated  with  SM.   Step  2:  Plan   Involve  the  user  community  via   focus  groups  to  incrementally  define   and  develop  the  SM  interven.on   plan.   Step  3:  Do   Flowing  from  CCS  needs  and  user   input,  implement  the  plan  and   finalize  the  evalua.on  research   framework  (data  collec.on   methods)  to  support  it.   Step  4:  Study   Collect  data  to  assess   implementa.on  progress  and   outcomes  and  regularly  report  to   CCS  on  results  to  inform  decision-­‐ making.   Development Cycle
  • 12. A mixed-methods approach to data collection was used to answer the following general questions: Developmental Evaluation Questions Question #1 – How did the Social media (SM) campaign develop over time and what evolved in its implementation?  Question #2 – What was learned in the pursuit of population reach using an SM-based tool with young adults?  Question #3 – How did the SM campaign change quitting success and influence Quitline utilization?  Question #4 – What influence did the social network have on quitting outcomes and youth engagement?
  • 13. Plan, Act, Evaluate Plan, Act, Evaluate Plan, Act, Evaluate “ready, fire, aim” approach to innovation Gamble, J. (2006). A Developmental Evaluation Primer Montreal, PQ: J.W.McConnell Foundation Developmental vs. Traditional Evaluation Approach Overlapping sequences of developments / evaluations
  • 14. 16   Step  2:  Plan   Involve  the  user  community   via  focus  groups  to   incrementally  define  and   develop  the  SM   interven.on  plan.   ROUND  I  focus  group  results:     Barriers  to  quiKng/triggers     Friends  who  smoke     Socializing  and  drinking     Rou.nes  that  include  smoking     Stress   Solu4ons     Change  rou.nes     Replace  with  healthy  habits     Support  of  friends   Reac4ons  to  Break  it  Off     Visually  appealing     Liked  the  “break-­‐up”  metaphor     Interac.ve  elements  important     App  was  well-­‐received   “…  it’s  kind  of  a  clever  idea,   the  whole  breaking  off  a   rela9onship  thing,  and  I   think  a  lot  of  people  will   relate  to  that.”   “I  think  that  one  major  factor   in  quiBng  smoking  is  to  just   know  that  you’re  not  alone  …   a  lot  of  friends  of  mine  quit   smoking  all  at  the  same   9me.”   “I  really  like  the  tracking   system  where  you  could  see   if  you  did  slip  up,  why  you   slipped  up  and  then  be  able   to  go  back  and  look  at  your   record  of  how  you  were   progressing  or  not   progressing.”  
  • 15. Data collection:   Minutes of meetings, SHL/SHO administrative data, and web analytic data collected.   11 young adult smoker focus group sessions (Ontario, Manitoba and New Brunswick) - 7 before the launch of the SM campaign and 4 after. Analysis:   Descriptive quantitative analysis of user activity.   Grounded Theory - Constant comparison method for qualitative focus group data. Measures:   Ongoing monitoring of decision- making, tracking of activity, SM usage and call volumes.   Young adult input on challenges to quitting, potential solutions, use of social media, and campaign feedback.
  • 16. Target population: English and French speaking smoking adults 19-29 years of age Outcome Measures:   Quitline Reach - proportion of eligible smokers in the target population who register for SHL/SHO - reach into the young adult smoking population is currently estimated to be just 0.5%. Data Collection and Analysis:   SHL/SHO administrative data on registrants and annual Canadian Tobacco Use Monitoring survey.   Reach seven months prior to the SM campaign launch and seven months calculated with sub-group comparisons. Reach * Effectiveness = Impact
  • 17. Sustainability of the SM Campaign supported by: Transla4on    An iterative, developmental approach,  Partnership with CCS using continuous knowledge exchange,  Embedding of the program within CCS’ suite of cessation programs 19   Step  5:  Act   In  collabora.on  with  CCS,  synthesis   and  transla.on  of  knowledge  into   publica.on,  programming  and/or   future  evalua.on  research.    
  • 18. Break  it  Off  -­‐  Real-­‐4me  Results   Number of people reached through the Break it Off web-site and level of activity  37,325 unique visitors  44,172 visits  107,600 page views  2.44 pages per visit on average  3,937 installations of the Break It Off smartphone app 20  
  • 19. Results  -­‐  SM  Performance   ROUND  II  focus  group  results  of  BIO   Users:       Two  focus  groups  conducted  to  date  (June  2012)     Site  and  app  were  easy  to  use,  visually  appealing,  and   suitable  for  audience     App  was  used  more  than  the  website     Using  app  to  track  progress  and  triggers  was  helpful     Mixed  percep.ons  on  SHL     Sharing  quit  status  on  Facebook  can  create  a  suppor.ve   environment   21   “I  posted  and  I  got  a  lot  of  good   feedback  on  it  so  that  helped   me  support-­‐wise  to  know  that   a  lot  of  people  actually  did  care   whether  I  quit  smoking.”   “The  app  helped  me  realize   what  my  triggers  were…so  then  I   could  stop  them  before  they   even  happened  and  just  kind  of   focus  on  something  else  or  keep   myself  busy  just  to  get  past  that   ini.al  craving  and  then  I  was   fine.”   “The  app  helped  me   stay  focused,  to  see  the   number  of  days  smoke-­‐ free  rise  every  day  was   key  for  me.”   Break  it  Off  -­‐  Real-­‐4me  Results  
  • 20. Did  web  traffic  increase  over  4me?   The  campaign  was  launched  on  Jan.  12,  2012  and  remained  fairly  constant  un.l   funding  for  promo.on  ran  out  at  the  end  of  March.  Spikes  in  traffic  can  be   aributed  to  promo.onal  ac.vi.es  such  as  the  home  page  takeover  of  the   Weather  Network.   •  Visits  per  day   22   Break  it  Off  -­‐  Real-­‐4me  Results  
  • 21. Where  were  users  geographically  located?       The  following  ci.es  provided  the     most  visits:   •  Toronto*  -­‐  8,363   •  Saskatoon*  -­‐  3,764   •  Regina*  -­‐  3,096   •  Oawa*  -­‐  2,288   •  Winnipeg    -­‐  1,400   •  Hamilton*  -­‐  1,229   *Provinces  with  expanded  promo.onal  BIO  campaigns   23   Break  it  Off  -­‐  Real-­‐4me  Results  
  • 22. Were  people  being  exposed  to  the  resources?       •  2,208  new  Facebook  page  likes  (www.facebook/smokershelpline)    since  January  2012   24   9,810   5,785   4,065   3,969   3,959   0   2,000  4,000  6,000  8,000  10,000  12,000   Break-­‐up  methods   Break-­‐up  videos   Smartphone  App   Get  Over  With  It   The  Ugly  Truth   Page  views  (Jan  12  to  Mar  31,  2012)   Break  it  Off  -­‐  Real-­‐4me  Results  
  • 23. Campaign  Results   25   Quit Rates 7 day PPA at 3 months – 47% 30 day PPA at 3 months – 32% Quit Actions Made a quit attempt – 91% Cut down amount smoked – 89% Set a quit date – 66% Satisfaction Very or mostly satisfied with the campaign – 77% Unintended Consequence Only 21 visitors out of 37,000+ connected to Quitline
  • 24. Results  -­‐  Reflec4on   Reflections on Developmental Evaluation Critical success factors   Having the right people involved   Partnership, open dialogue, relationships   Expertise in social media   Goal congruence   Adaptable and agile   Social Innovative nature was appealing   Enthusiasm for the project 26  
  • 25. Lessons  Learned  &  Next  Steps   Lessons Learned:   Flexibility and adaptation are key   “It’s about relationships”   Continuous learning   Organizations are not set up for development, just outcomes… development-in-action is challenging 27  
  • 26. Conclusion   28   Implications for Practice   Break It Off was effective in helping young adults quit smoking.   Compares favourably to other digital social media interventions - “Happy Endings” 44.7% quit rate (7 day PPA) versus “Break it off” 47% quit rate (7 day PPA) at 3 months   Break It Off is reaching smokers who are not using cessation supports such as SHL   An integrated tobacco cessation system should consider the role of social media based interventions and the implications of technological change on service offerings
  • 27. Conclusion   29   Implications for Research & Evaluation   Formative evaluation research on social media - e.g. What seems to be working, not working, reasons for use, etc.?   Implementation science - e.g. What are the key components that account for change in behaviour for social media?   Paucity of evidence on effectiveness and cost- effectiveness of social media interventions for decision-makers.
  • 28. Cameron D. Norman PhD CENSE Research + Design cdnorman@cense.ca @cdnorman N. Bruce Baskerville PhD Propel Centre for Population Health Impact, University of Waterloo nbbasker@uwaterloo.ca

×