Evaluation of Break It Off:
a Social Media-based Campaign
Targeting Young Adult Smokers
Cameron D. Norman1
N.Bruce Baskervi...
BACKGROUND
  Young adults (19-29) have the highest rate of smoking in
Canada – 24.4% use Tobacco (CCHS, 2010)
  86% of young adults...
Any electronic, networked information resource
that derives its principal value from user
contributions, engagement & inte...
Developmental Evaluation
Developmental Evaluation
•  First proposed by Michael Quinn Patton
(1994)
•  Combines Utilization-Focused
Evaluation with ...
Developmental Evaluation
•  Assists social innovators to learn
strategically
•  Supports adaptation to changing
conditions...
Systems Thinking
•  Knowledge structures can be built upon:
–  Information: symbolic pattern of matter and
energy
–  Netwo...
Break	
  It	
  Off	
  
11	
  
  Promoted January 2012 to March 2012
•  Paid advertising on websites with high levels of yo...
What are Quitlines?
  Quitlines are computer-supported, telephone-based
tobacco cessation counselling services that help ...
Developmental	
  Evalua4on	
  Research	
  	
  
Approach	
  
13	
  
Step	
  5:	
  Act	
  
In	
  collabora.on	
  with	
  CCS...
A mixed-methods approach to
data collection was used to
answer the following general
questions:
Developmental
Evaluation Q...
Plan, Act, Evaluate
Plan, Act, Evaluate
Plan, Act, Evaluate
“ready, fire, aim” approach to innovation
Gamble, J. (2006). A ...
16	
  
Step	
  2:	
  Plan	
  
Involve	
  the	
  user	
  community	
  
via	
  focus	
  groups	
  to	
  
incrementally	
  de...
Data collection:
  Minutes of meetings, SHL/SHO administrative data, and web
analytic data collected.
  11 young adult s...
Target population: English and French
speaking smoking adults 19-29 years of
age
Outcome Measures:
  Quitline Reach - pro...
Sustainability of the SM
Campaign supported by:
Transla4on	
  
 An iterative, developmental approach,
 Partnership with ...
Break	
  it	
  Off	
  -­‐	
  Real-­‐4me	
  Results	
  
Number of people reached through the
Break it Off web-site and level ...
Results	
  -­‐	
  SM	
  Performance	
  
ROUND	
  II	
  focus	
  group	
  results	
  of	
  BIO	
  
Users:	
  	
  
  Two	
 ...
Did	
  web	
  traffic	
  increase	
  over	
  4me?	
  
The	
  campaign	
  was	
  launched	
  on	
  Jan.	
  12,	
  2012	
  and...
Where	
  were	
  users	
  geographically	
  located?	
  	
  	
  
The	
  following	
  ci.es	
  provided	
  the	
  	
  
most...
Were	
  people	
  being	
  exposed	
  to	
  the	
  resources?	
  	
  	
  
•  2,208	
  new	
  Facebook	
  page	
  likes	
  ...
Campaign	
  Results	
  
25	
  
Quit Rates
7 day PPA at 3 months – 47%
30 day PPA at 3 months – 32%
Quit Actions
Made a qui...
Results	
  -­‐	
  Reflec4on	
  
Reflections on Developmental
Evaluation
Critical success factors
  Having the right people ...
Lessons	
  Learned	
  &	
  Next	
  Steps	
  
Lessons Learned:
  Flexibility and adaptation are key
  “It’s about relatio...
Conclusion	
  
28	
  
Implications for Practice
  Break It Off was effective in helping young adults
quit smoking.
  Compa...
Conclusion	
  
29	
  
Implications for Research & Evaluation
  Formative evaluation research on social media -
e.g. What ...
Cameron D. Norman PhD
CENSE Research + Design
cdnorman@cense.ca
@cdnorman
N. Bruce Baskerville PhD
Propel Centre for Popul...
BreakItOff Evaluation
BreakItOff Evaluation
Upcoming SlideShare
Loading in …5
×

BreakItOff Evaluation

1,029 views

Published on

Presentation delivered at the Canadian Evaluation Society Conference in Toronto in June 2013. The results and process behind the development evaluation and outcome evaluation with the Break It Off social media campaign for youth smoking cessation were discussed.

Published in: Health & Medicine, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,029
On SlideShare
0
From Embeds
0
Number of Embeds
85
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

BreakItOff Evaluation

  1. 1. Evaluation of Break It Off: a Social Media-based Campaign Targeting Young Adult Smokers Cameron D. Norman1 N.Bruce Baskerville2 ,Nancy Korstanje3 Steve Brown2, Barbara Zupko2 ,Larry Squire2 1 CENSE Research + Design & Dalla Lana School of Public Health, University of Toronto 2Propel Centre for Population Health Impact, 3The Canadian Cancer Society Canadian Evaluation Society Conference, Toronto, ON June 2013
  2. 2. BACKGROUND
  3. 3.   Young adults (19-29) have the highest rate of smoking in Canada – 24.4% use Tobacco (CCHS, 2010)   86% of young adults (18 to 34 years) in Canada are using social media tools – e.g Facebook, Twitter, etc. (Ipsos Reid, 2011)   Social media tools are immensely popular, yet not well studied in terms of population health impact   Evidence to-date suggest wide reach, low barriers to use, and portability → great potential for population level impact     3  
  4. 4. Any electronic, networked information resource that derives its principal value from user contributions, engagement & interaction
  5. 5. Developmental Evaluation
  6. 6. Developmental Evaluation •  First proposed by Michael Quinn Patton (1994) •  Combines Utilization-Focused Evaluation with concepts from Complexity Science •  Social innovation and resiliency require actionable feedback to enable programs to adapt and thrive through change •  Strategic learning / real-time evaluation
  7. 7. Developmental Evaluation •  Assists social innovators to learn strategically •  Supports adaptation to changing conditions •  Develops promising interventions •  Documents innovators’ actions •  Identifies emergent processes and outcomes; enhances resiliency •  Determines if an innovation is ready for formative evaluation
  8. 8. Systems Thinking •  Knowledge structures can be built upon: –  Information: symbolic pattern of matter and energy –  Networks: the information delivery channels and attendant limits –  Systems: Bounded elements linked to facilitate the flow of information and support networks
  9. 9. Break  It  Off   11     Promoted January 2012 to March 2012 •  Paid advertising on websites with high levels of young adult viewership. •  Banner ads on sites such as Microsoft, Yahoo and Facebook. •  Homepage takeovers on the Weather Network. •  Ads on dating sites such as Plenty of Fish and OK Cupid. •  Facebook, Twitter, YouTube videos and blogs.   Links to Smokers’ Helpline
  10. 10. What are Quitlines?   Quitlines are computer-supported, telephone-based tobacco cessation counselling services that help tobacco users quit involving a live, trained cessation counsellor.   Quitlines reach and serve tobacco users, regardless of location.   Residents in all 50 states, the District of Columbia, each U.S. territory, all ten Canadian provinces and two territories have access to public Quitline services.   Smokers’ Helpline (SHL) is an evidence-based cessation service managed by the Canadian Cancer Society.   SHL reach into the young adult smoking population is estimated to be just 0.5%. 12  
  11. 11. Developmental  Evalua4on  Research     Approach   13   Step  5:  Act   In  collabora.on  with  CCS,  synthesis   and  transla.on  of  knowledge  into   publica.on,  programming  and/or   future  evalua.on  research.     Step  1:  Listen   Interact  with  CCS  to  iden.fy  their   needs  and  wants  and  build  a  shared   understanding  of  the  context,   objec.ves,  challenges  and   opportuni.es  associated  with  SM.   Step  2:  Plan   Involve  the  user  community  via   focus  groups  to  incrementally  define   and  develop  the  SM  interven.on   plan.   Step  3:  Do   Flowing  from  CCS  needs  and  user   input,  implement  the  plan  and   finalize  the  evalua.on  research   framework  (data  collec.on   methods)  to  support  it.   Step  4:  Study   Collect  data  to  assess   implementa.on  progress  and   outcomes  and  regularly  report  to   CCS  on  results  to  inform  decision-­‐ making.   Development Cycle
  12. 12. A mixed-methods approach to data collection was used to answer the following general questions: Developmental Evaluation Questions Question #1 – How did the Social media (SM) campaign develop over time and what evolved in its implementation?  Question #2 – What was learned in the pursuit of population reach using an SM-based tool with young adults?  Question #3 – How did the SM campaign change quitting success and influence Quitline utilization?  Question #4 – What influence did the social network have on quitting outcomes and youth engagement?
  13. 13. Plan, Act, Evaluate Plan, Act, Evaluate Plan, Act, Evaluate “ready, fire, aim” approach to innovation Gamble, J. (2006). A Developmental Evaluation Primer Montreal, PQ: J.W.McConnell Foundation Developmental vs. Traditional Evaluation Approach Overlapping sequences of developments / evaluations
  14. 14. 16   Step  2:  Plan   Involve  the  user  community   via  focus  groups  to   incrementally  define  and   develop  the  SM   interven.on  plan.   ROUND  I  focus  group  results:     Barriers  to  quiKng/triggers     Friends  who  smoke     Socializing  and  drinking     Rou.nes  that  include  smoking     Stress   Solu4ons     Change  rou.nes     Replace  with  healthy  habits     Support  of  friends   Reac4ons  to  Break  it  Off     Visually  appealing     Liked  the  “break-­‐up”  metaphor     Interac.ve  elements  important     App  was  well-­‐received   “…  it’s  kind  of  a  clever  idea,   the  whole  breaking  off  a   rela9onship  thing,  and  I   think  a  lot  of  people  will   relate  to  that.”   “I  think  that  one  major  factor   in  quiBng  smoking  is  to  just   know  that  you’re  not  alone  …   a  lot  of  friends  of  mine  quit   smoking  all  at  the  same   9me.”   “I  really  like  the  tracking   system  where  you  could  see   if  you  did  slip  up,  why  you   slipped  up  and  then  be  able   to  go  back  and  look  at  your   record  of  how  you  were   progressing  or  not   progressing.”  
  15. 15. Data collection:   Minutes of meetings, SHL/SHO administrative data, and web analytic data collected.   11 young adult smoker focus group sessions (Ontario, Manitoba and New Brunswick) - 7 before the launch of the SM campaign and 4 after. Analysis:   Descriptive quantitative analysis of user activity.   Grounded Theory - Constant comparison method for qualitative focus group data. Measures:   Ongoing monitoring of decision- making, tracking of activity, SM usage and call volumes.   Young adult input on challenges to quitting, potential solutions, use of social media, and campaign feedback.
  16. 16. Target population: English and French speaking smoking adults 19-29 years of age Outcome Measures:   Quitline Reach - proportion of eligible smokers in the target population who register for SHL/SHO - reach into the young adult smoking population is currently estimated to be just 0.5%. Data Collection and Analysis:   SHL/SHO administrative data on registrants and annual Canadian Tobacco Use Monitoring survey.   Reach seven months prior to the SM campaign launch and seven months calculated with sub-group comparisons. Reach * Effectiveness = Impact
  17. 17. Sustainability of the SM Campaign supported by: Transla4on    An iterative, developmental approach,  Partnership with CCS using continuous knowledge exchange,  Embedding of the program within CCS’ suite of cessation programs 19   Step  5:  Act   In  collabora.on  with  CCS,  synthesis   and  transla.on  of  knowledge  into   publica.on,  programming  and/or   future  evalua.on  research.    
  18. 18. Break  it  Off  -­‐  Real-­‐4me  Results   Number of people reached through the Break it Off web-site and level of activity  37,325 unique visitors  44,172 visits  107,600 page views  2.44 pages per visit on average  3,937 installations of the Break It Off smartphone app 20  
  19. 19. Results  -­‐  SM  Performance   ROUND  II  focus  group  results  of  BIO   Users:       Two  focus  groups  conducted  to  date  (June  2012)     Site  and  app  were  easy  to  use,  visually  appealing,  and   suitable  for  audience     App  was  used  more  than  the  website     Using  app  to  track  progress  and  triggers  was  helpful     Mixed  percep.ons  on  SHL     Sharing  quit  status  on  Facebook  can  create  a  suppor.ve   environment   21   “I  posted  and  I  got  a  lot  of  good   feedback  on  it  so  that  helped   me  support-­‐wise  to  know  that   a  lot  of  people  actually  did  care   whether  I  quit  smoking.”   “The  app  helped  me  realize   what  my  triggers  were…so  then  I   could  stop  them  before  they   even  happened  and  just  kind  of   focus  on  something  else  or  keep   myself  busy  just  to  get  past  that   ini.al  craving  and  then  I  was   fine.”   “The  app  helped  me   stay  focused,  to  see  the   number  of  days  smoke-­‐ free  rise  every  day  was   key  for  me.”   Break  it  Off  -­‐  Real-­‐4me  Results  
  20. 20. Did  web  traffic  increase  over  4me?   The  campaign  was  launched  on  Jan.  12,  2012  and  remained  fairly  constant  un.l   funding  for  promo.on  ran  out  at  the  end  of  March.  Spikes  in  traffic  can  be   aributed  to  promo.onal  ac.vi.es  such  as  the  home  page  takeover  of  the   Weather  Network.   •  Visits  per  day   22   Break  it  Off  -­‐  Real-­‐4me  Results  
  21. 21. Where  were  users  geographically  located?       The  following  ci.es  provided  the     most  visits:   •  Toronto*  -­‐  8,363   •  Saskatoon*  -­‐  3,764   •  Regina*  -­‐  3,096   •  Oawa*  -­‐  2,288   •  Winnipeg    -­‐  1,400   •  Hamilton*  -­‐  1,229   *Provinces  with  expanded  promo.onal  BIO  campaigns   23   Break  it  Off  -­‐  Real-­‐4me  Results  
  22. 22. Were  people  being  exposed  to  the  resources?       •  2,208  new  Facebook  page  likes  (www.facebook/smokershelpline)    since  January  2012   24   9,810   5,785   4,065   3,969   3,959   0   2,000  4,000  6,000  8,000  10,000  12,000   Break-­‐up  methods   Break-­‐up  videos   Smartphone  App   Get  Over  With  It   The  Ugly  Truth   Page  views  (Jan  12  to  Mar  31,  2012)   Break  it  Off  -­‐  Real-­‐4me  Results  
  23. 23. Campaign  Results   25   Quit Rates 7 day PPA at 3 months – 47% 30 day PPA at 3 months – 32% Quit Actions Made a quit attempt – 91% Cut down amount smoked – 89% Set a quit date – 66% Satisfaction Very or mostly satisfied with the campaign – 77% Unintended Consequence Only 21 visitors out of 37,000+ connected to Quitline
  24. 24. Results  -­‐  Reflec4on   Reflections on Developmental Evaluation Critical success factors   Having the right people involved   Partnership, open dialogue, relationships   Expertise in social media   Goal congruence   Adaptable and agile   Social Innovative nature was appealing   Enthusiasm for the project 26  
  25. 25. Lessons  Learned  &  Next  Steps   Lessons Learned:   Flexibility and adaptation are key   “It’s about relationships”   Continuous learning   Organizations are not set up for development, just outcomes… development-in-action is challenging 27  
  26. 26. Conclusion   28   Implications for Practice   Break It Off was effective in helping young adults quit smoking.   Compares favourably to other digital social media interventions - “Happy Endings” 44.7% quit rate (7 day PPA) versus “Break it off” 47% quit rate (7 day PPA) at 3 months   Break It Off is reaching smokers who are not using cessation supports such as SHL   An integrated tobacco cessation system should consider the role of social media based interventions and the implications of technological change on service offerings
  27. 27. Conclusion   29   Implications for Research & Evaluation   Formative evaluation research on social media - e.g. What seems to be working, not working, reasons for use, etc.?   Implementation science - e.g. What are the key components that account for change in behaviour for social media?   Paucity of evidence on effectiveness and cost- effectiveness of social media interventions for decision-makers.
  28. 28. Cameron D. Norman PhD CENSE Research + Design cdnorman@cense.ca @cdnorman N. Bruce Baskerville PhD Propel Centre for Population Health Impact, University of Waterloo nbbasker@uwaterloo.ca

×