Evaluation
Background <ul><li>Positive Life has developed a broad based social marketing campaign that aimed to inform both HIV posit...
Underlying values and principles   <ul><li>Campaign development is evidence based with social research data and community ...
Underlying values and principles   <ul><li>Issues </li></ul><ul><li>Research found that in the case of regular partners th...
Overall campaign aims <ul><li>Normalise serodiscordant relationships and challenge HIV stigma </li></ul><ul><li>Support ga...
Overall campaign aims <ul><li>Support the practice of testing for HIV and STIs as part of negotiating safe sex within the ...
Feb 2009 April 2009 Feb 2010 Three community forums hosted by Vanessa Wagner (format changed each time) Dinner Forum 2,500...
Campaign materials for testing
(The   ‘National Evaluation Framework for HIV/AIDS Health Promotion for Gay and Other Homosexually Active Men’ framed the ...
Evaluation questions <ul><li>Key evaluation questions included: </li></ul><ul><li>How was the campaign implemented? </li><...
Evaluation Components: 1.  Online survey (GAYDAR) which aimed to  understand who the campaign reached and  how well it com...
Campaign Design
The campaign was designed to use the imagery and language of the target audience. Images and language that men in or think...
The campaign was designed to use the imagery  and language of the target audience <ul><li>Designed to be realistic  </li><...
<ul><li>About  respondents </li></ul>
 
 
Respondents (summary)  <ul><li>256 survey respondents (160 respondents  (63%) were from NSW) </li></ul><ul><li>Age - large...
 
Evaluation Outcomes
Evaluation Outcomes Only 160 respondents from NSW are included in the analysis of image recall (no messages or logos were ...
Evaluation Outcomes 132 51 NSW respondents who answered question This result is not surprising given we only placed this a...
Evaluation Outcomes   174 answered question 6% Irrelevant or general comments 2% Supporting serodiscordant relationships 2...
Evaluation Outcomes As this advertisement contains specific and potentially new information to the  audience, respondents ...
Evaluation Outcomes  (Viral Load Ad) <ul><li>70% of all respondents believe this advertisement addresses a relevant issue;...
194 answered question 27.8% Boring 27.8% Good way to make a point 5.7% Seen it all before 21.6% Gets your attention 17.0% ...
 
Other responses   <ul><li>“ If the ads are meant to promote serodiscordant relationships, they fail because they do not ad...
Impact
Impact  <ul><li>53% of respondents strongly agree or agree that the advertisements raised their awareness of the issues fa...
SERO DISCO Magazine - Survey Responses 143 72 NSW and Victoria respondents who answered question  6.3% 5.6% Unsure 71.3% 6...
SERO DISCO Magazine - Survey Responses 37 27 answered question 18.9% 25.9 Did not really look at it 5.4% 7.4 Look up speci...
SERO DISCO Magazine - Survey Responses 37 answered question 2.7% Information stall 5.4% Community event 10.8% Gay venue 13...
 
SERO  DISCO  Magazine – focus group responses <ul><li>Participants  </li></ul><ul><li>found the magazine an informative an...
SERO DISCO Magazine- focus group responses <ul><li>they also reported a more positive response to some of the  </li></ul><...
Conclusions <ul><li>Recall - </li></ul><ul><li>Time lapse from the end of the campaign until the evaluation (ten months) w...
Conclusion <ul><li>SERO DISCO Ads -  </li></ul><ul><li>Respondents were asked to indicate which answer best  </li></ul><ul...
Conclusion <ul><li>SERO DISCO Magazine - </li></ul><ul><li>Received a favourable (positive) response </li></ul><ul><li>Par...
Conclusion <ul><li>SERO DISCO Magazine – </li></ul><ul><li>Challenge is ensuring it is picked up: </li></ul><ul><li>Engagi...
Conclusion <ul><li>Survey respondents confirmed the already known shift towards online social networking  sites (such as F...
Lessons Learned  <ul><li>Life stories produced in a magazine format and community based social/ peer education events have...
Lessons Learned  <ul><li>Evaluation  </li></ul><ul><li>- needs to consider all campaign components and  </li></ul><ul><li>...
<ul><li>Acknowledgements: </li></ul><ul><li>Aldo Spina – Evaluation Consultant </li></ul><ul><li>HIV/AIDS & Related Progra...
Upcoming SlideShare
Loading in...5
×

Evaluation

407

Published on

Presentation by Kathy Triffit, Positive Life NSW, at the 2010 AFAO HIV Educators Conference.

Published in: Health & Medicine
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
407
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Forum aims: Act as a catalyst to network gay men in serodiscordant relationships. Feed into existing social networks and ongoing support mechanisms. Provide an opportunity for men in or thinking about a serodiscordant relationship to explore issues like managing risk and communication, starting and maintaining a relationship, disclosure, condoms, relationship agreements, testing for HIV, PEP, STI awareness and more. Share their experiences so participants can learn skills to manage the challenges and issues they encounter in their relationships.
  • Of the forty-six NSW respondents who stated that they recalled the images, ten respondents were able to recall (unprompted) messages that accompanied the images. Of these, nine successfully identified that it related to serodiscordant relationships. Two of these respondents closely recalled the campaign slogan. Another more generally (and not incorrectly) recalled that they thought the campaign was regarding safe sex. Respondents who recalled the images were asked if they could recall the organisation associated with the advertisements. Thirteen respondents recall ACON, eight others correctly recalled Positive Life NSW. If you exclude staff/volunteers, twelve respondents recalled ACON and four Positive Life. Respondents were asked where they had seen the images. Most frequently they reported they had seen it in the gay media. Other places where they recalled seeing the images were at their doctors or sexual health clinics, Talkabout, or gay venues. One-fifth of respondents indicated that they had seen it on the Positive Life website or at a community organisation.
  • Second advertisement
  • Sixty-five per cent of respondents identify that the key message of the advertisement is to use condoms. However, another third of the sample had different interpretations. Nine per cent of respondents interpret the key message as saying that if you have undetectable viral load it reduces the risk of HIV transmission and therefore, they concluded, its ok not to use condoms. Another nine per cent of respondents were unsure or confused about what the key message was. Respondents said it is “confusing” and “gives mixed messages”. Others simply said they “had no idea”, “not sure” or “dunno”. Three respondents indicated that they thought one of the key messages of the advertisement was to support HIV serodiscordant couples.
  • The most common word chosen was that there was nothing special about the cartoons. Thirty-seven per cent of respondents chose this word. Slightly less, 33 per cent, said they were ok. Twenty-eight per cent thought they were boring. Other participants had a positive response. Twenty-eight per cent thought they were a good way to make a point, 26 per cent of respondents indicated that they made them laugh and 27 per cent also thought they were cute. Twenty-one per cent of respondents thought they get your attention. An equal number of respondents, 17 per cent each, like and dislike cartoons.
  • Next SLIDE
  • Evaluation

    1. 1. Evaluation
    2. 2. Background <ul><li>Positive Life has developed a broad based social marketing campaign that aimed to inform both HIV positive and negative gay men (who are in or thinking about a serodiscordant relationship) on strategies to maintain their and their partners’ health. </li></ul>
    3. 3. Underlying values and principles <ul><li>Campaign development is evidence based with social research data and community consultations (community knowledges) providing the background to this work. </li></ul><ul><li>Gay men in serodiscordant relationships were invited to take part in the development of campaign content and imagery. This included discussion groups, photo sessions, community forums and interviews. (We looked at topics such as pleasure and intimacy, disclosure, risk assessment and management, viral load and </li></ul><ul><li>infectivity, attitudes to condoms and so on). </li></ul>
    4. 4. Underlying values and principles <ul><li>Issues </li></ul><ul><li>Research found that in the case of regular partners the first six months in serodiscordant relationships are when new HIV infections are more likely to occur. </li></ul><ul><li>A number of studies* also report on how gay men in these relationships are using undetectable viral load to reduce their risk of passing on or getting HIV. </li></ul><ul><li>* Jin. F., Prestage, G.P., Ellard, J., Kippax, S.C., Kaldor, J.M., Grulich, A.E., (2007) ‘How homosexual men believe they </li></ul><ul><li>became infected with HIV: the role of risk-reduction behaviours’, Journal of Acquired Immune Deficiency Syndromes: </li></ul><ul><li>JAIDS, 46(2):245-7 </li></ul>
    5. 5. Overall campaign aims <ul><li>Normalise serodiscordant relationships and challenge HIV stigma </li></ul><ul><li>Support gay men entering into new </li></ul><ul><li>serodiscordant relationships to maintain </li></ul><ul><li>a commitment to safe sex and to reinforce </li></ul><ul><li>the effectiveness of safe sex practices </li></ul><ul><li>Offer communication and negotiation </li></ul><ul><li>strategies, and other health information, </li></ul><ul><li>which will support the prevention of HIV/STIs </li></ul><ul><li>and a better understanding of sexual health needs </li></ul>
    6. 6. Overall campaign aims <ul><li>Support the practice of testing for HIV and STIs as part of negotiating safe sex within the relationship </li></ul><ul><li>Develop a response to discussions on the Swiss Consensus Statement to give gay men in serodiscordant relationships clearer information on viral load and infectivity. </li></ul>
    7. 7. Feb 2009 April 2009 Feb 2010 Three community forums hosted by Vanessa Wagner (format changed each time) Dinner Forum 2,500 Campaign t-shirts Distributed at first dinner forum and Mardi Gras Fair Day 2009 T shirts 15/12/08 - 15/02/09 Gaydar (Sydney and NSW only) Banner ads Full page ads were produced as quarter page ads Three strip ads each with a different cartoon strip Quarter page ads, half page ads, strip ads 2,500 Each with a different cartoon strip Four postcards Ad 1: Why let HIV get in the way of good relationship Ad 2: Disclosure Ad 3: Undetectable viral load Ad 4: HIV and STI Testing Full pages ads 6,000 NSW 3,000 Vic 48 glossy colour magazine features range of couples reflecting on their experiences. Articles on a broad range of relationship and HIV issues. Sero Disco Magazine Campaign components
    8. 8. Campaign materials for testing
    9. 9. (The ‘National Evaluation Framework for HIV/AIDS Health Promotion for Gay and Other Homosexually Active Men’ framed the focus of the evaluation.) <ul><li>The evaluation focused on process and impact. </li></ul><ul><li>Process evaluation measures processes, </li></ul><ul><li>activities and methods. </li></ul><ul><li>Impact evaluation measures the immediate </li></ul><ul><li>effects of the health promotion activity. </li></ul>Evaluation approach
    10. 10. Evaluation questions <ul><li>Key evaluation questions included: </li></ul><ul><li>How was the campaign implemented? </li></ul><ul><li>What was the reach of the campaign? </li></ul><ul><li>Are clients satisfied with the campaign? </li></ul><ul><li>What impact did the campaign have on </li></ul><ul><li>heath literacy? </li></ul><ul><li>Did the campaign achieve its objectives? </li></ul>
    11. 11. Evaluation Components: 1. Online survey (GAYDAR) which aimed to understand who the campaign reached and how well it communicated its key messages. 2. Focus groups (x 2) which aimed to assess: - campaign satisfaction, including whether the messages were understood and accepted [and] - the immediate impact the messages had on awareness and knowledge.
    12. 12. Campaign Design
    13. 13. The campaign was designed to use the imagery and language of the target audience. Images and language that men in or thinking about a serodiscordant relationship identify with ?
    14. 14. The campaign was designed to use the imagery and language of the target audience <ul><li>Designed to be realistic </li></ul><ul><li>Used narratives (personal experiences and photos) to reflect the experiences of men within the target audience </li></ul><ul><li>Used humour (cartoons to get some of the messages across) </li></ul>Language
    15. 15. <ul><li>About respondents </li></ul>
    16. 18. Respondents (summary) <ul><li>256 survey respondents (160 respondents (63%) were from NSW) </li></ul><ul><li>Age - largest group of respondents 40 – 49, </li></ul><ul><li>followed closely by 30 -39. </li></ul><ul><li>- quarter of the respondents were aged 29 and </li></ul><ul><li>under </li></ul><ul><li>People with HIV tended to be older with 60% aged </li></ul><ul><li>between 40 – 59 </li></ul><ul><li>Majority of respondents were HIV negative (54.4%) </li></ul><ul><li>People with HIV constituted 34% (86) of all </li></ul><ul><li>respondents (with 59 living in NSW) </li></ul>
    17. 20. Evaluation Outcomes
    18. 21. Evaluation Outcomes Only 160 respondents from NSW are included in the analysis of image recall (no messages or logos were shown). 29 % (46 respondents) recall is influenced by the time lag (approx 10 months) between ads finishing and the survey commencing. 9.5% 8.8% Unsure 61.4% 43.9% No 29.1% 47.4% (46) Yes NSW responses People with HIV in NSW Answer options Have you seen the above images used in an advertisement?
    19. 22. Evaluation Outcomes 132 51 NSW respondents who answered question This result is not surprising given we only placed this ad once in SSO and Talkabout 14.4% 11.8% Unsure 66.7% 60.8% No 18.9% 27.5% Yes NSW responses People with HIV in NSW Answer options Do you recall having seen this advertisement before?
    20. 23. Evaluation Outcomes 174 answered question 6% Irrelevant or general comments 2% Supporting serodiscordant relationships 2% Communicate with one another 7% Other message 21% Undetectable viral load does not mean no risk, still use condoms 9% Undetectable viral load reduces risk, ok not to use condoms 9% Unsure or confused 14% Practice safe sex 30% Use condoms and lube Response Answers What would you say is its key message?
    21. 24. Evaluation Outcomes As this advertisement contains specific and potentially new information to the audience, respondents were asked if the advertisement had increased their understanding of the risk of using undetectable viral load to negotiate sex.
    22. 25. Evaluation Outcomes (Viral Load Ad) <ul><li>70% of all respondents believe this advertisement addresses a relevant issue; 57% believe it provides useful information </li></ul><ul><li>Where problems potentially arise is how effective it communicates its message. 46% strongly agree or agree that it does communicate its message effectively, however 25% are neutral and another 29% disagree or strongly agree </li></ul><ul><li>The response is similar when respondents are asked if it’s confusing (e.g. in this case sends out the wrong message). 36% strongly agree or agree it’s confusing, 17% are neutral and 46% disagree or strongly agree </li></ul><ul><li>The response was more positive when respondents were asked if the ad was easy to understand . 58% strongly agree or agree, 16% were neutral and 27% disagree or strongly agree. </li></ul>
    23. 26. 194 answered question 27.8% Boring 27.8% Good way to make a point 5.7% Seen it all before 21.6% Gets your attention 17.0% I like cartoons 27.3% Cute 17.5% I don’t like cartoons 37.1% Nothing special 32.5% Ok 26.3% Made me laugh Response Answer options (multiple answers permitted) What do you think of the cartoons used in the advertisements above?
    24. 28. Other responses <ul><li>“ If the ads are meant to promote serodiscordant relationships, they fail because they do not address the true fears of neg men confronting the possibility of a relationship with a pos male. Also unrealistic because a condom will not address neg fears.” </li></ul><ul><li>Ten respondents made some comment on the “confusing” message or concerns that Positive Life was sending out the “wrong message” re the viral load ad. </li></ul><ul><li>“ In my opinion, it implicitly condones unprotected sex IF viral load is undetectable for 6 months and meds are never forgotten.” </li></ul>
    25. 29. Impact
    26. 30. Impact <ul><li>53% of respondents strongly agree or agree that the advertisements raised their awareness of the issues facing gay men in serodiscordant relationships. </li></ul><ul><li>50% said they strongly agree or agree that they have become more aware of the issues facing people with HIV </li></ul><ul><li>50% of all respondents strongly agree or agree that the advertisements informed them of the risks of using undetectable viral load when negotiating sex (while a few of the respondents stated they did not know what undetectable viral load means or found the message confusing ) </li></ul><ul><li>27% strongly agree or agree that the campaign will not have any impact, and another 27% are neutral. </li></ul>
    27. 31. SERO DISCO Magazine - Survey Responses 143 72 NSW and Victoria respondents who answered question 6.3% 5.6% Unsure 71.3% 63.9% No 22.4% 30.6% Yes Response - All People with HIV Answer options Do you recall seeing this magazine?
    28. 32. SERO DISCO Magazine - Survey Responses 37 27 answered question 18.9% 25.9 Did not really look at it 5.4% 7.4 Look up specific information 21.6% 14.8 Read most of it 21.6% 18.5 Read a fair bit of it 32.4% 33.3 Peruse the magazine Response - All Response Answer options You indicated that you recall seeing the &quot;Sero Disco&quot; magazine. Did you:
    29. 33. SERO DISCO Magazine - Survey Responses 37 answered question 2.7% Information stall 5.4% Community event 10.8% Gay venue 13.5% Other health services and/or community organisations 29.7% AIDS Council, GLBT organisation, HIV organisation 10.8% Sexual health clinic 18.9% Doctors 8.1% Don’t remember Response Answer options Do you recall where you first saw the magazine?
    30. 35. SERO DISCO Magazine – focus group responses <ul><li>Participants </li></ul><ul><li>found the magazine an informative and interesting read </li></ul><ul><li>liked the relevance of issues addressed, as well as the way it addressed HIV issues in a broader context </li></ul><ul><li>thought the format worked well, covered the topics in more depth (e.g. viral load was able to be more appropriately addressed) </li></ul><ul><li>liked photographs of ‘real people’ and the ‘diversity of couples’ represented </li></ul>
    31. 36. SERO DISCO Magazine- focus group responses <ul><li>they also reported a more positive response to some of the </li></ul><ul><li>advertisements because they were more stripped back so </li></ul><ul><li>there was only the cartoon image. </li></ul>
    32. 37. Conclusions <ul><li>Recall - </li></ul><ul><li>Time lapse from the end of the campaign until the evaluation (ten months) will have had a negative impact upon recall (therefore too much should not be made of the low level of recall) </li></ul><ul><li>Relevance - </li></ul><ul><li>Half of the survey respondents reported that the campaign made them more aware of the issues facing people with HIV. </li></ul><ul><li>They also reported that it raised their awareness of the issues facing gay men in serodiscordant relationships. </li></ul>
    33. 38. Conclusion <ul><li>SERO DISCO Ads - </li></ul><ul><li>Respondents were asked to indicate which answer best </li></ul><ul><li>reflects their overall response to the advertisements: </li></ul><ul><li>40% indicated that they have mixed feelings. They like some elements but dislike others. </li></ul><ul><li>34% think they are good advertisements. And </li></ul><ul><li>another quarter think they are nothing special, </li></ul><ul><li>just another safe sex campaign. </li></ul><ul><li>Cartoons (humour) versus real stories and photographs? </li></ul>
    34. 39. Conclusion <ul><li>SERO DISCO Magazine - </li></ul><ul><li>Received a favourable (positive) response </li></ul><ul><li>Participants responded well to the magazine format because </li></ul><ul><li>of its capacity to: </li></ul><ul><li>- address issues in more depth (e.g. viral load) </li></ul><ul><li>- include a blend of lived experiences and ways to enhance </li></ul><ul><li>quality of life outside of just talking about negotiating sex </li></ul><ul><li>Preferred the focus on stories and photographs of real people </li></ul><ul><li>Articles interesting and relevant </li></ul><ul><li>Among survey respondents only 37 in total recall seeing the magazine, which makes it hard to draw definitive conclusions. Of these, half only perused the magazine. Half of all respondents did think it was an enjoyable read, another </li></ul><ul><li>quarter thought it was an ok read. </li></ul>
    35. 40. Conclusion <ul><li>SERO DISCO Magazine – </li></ul><ul><li>Challenge is ensuring it is picked up: </li></ul><ul><li>Engaging or making the magazine relevant </li></ul><ul><li>to a broader range of HIV negative men would </li></ul><ul><li>mean re-branding the magazine. </li></ul><ul><li>Other challenges: </li></ul><ul><li>How do we engage HIV negative gay men in HIV? </li></ul><ul><li>How do we engage the attention of the target audience “bombarded with advertising everywhere”, </li></ul><ul><li>“ just another safe sex campaign”? </li></ul>
    36. 41. Conclusion <ul><li>Survey respondents confirmed the already known shift towards online social networking sites (such as Facebook- 39% of respondents) and hooking up websites (such as Gaydar- 54% of respondents). </li></ul><ul><li>Gay venues remain important places to promote materials with 62% of respondents visiting at least every few months. </li></ul><ul><li>Half of the respondents never actively engage with gay/ HIV community organisations; 30% actively engage with such organisations fairly regularly. </li></ul>
    37. 42. Lessons Learned <ul><li>Life stories produced in a magazine format and community based social/ peer education events have the capacity to invite discussion, challenge HIV stigma and ‘normalise’ serodiscordant relationships for both HIV positive and negative gay men. </li></ul><ul><li>Education and support are long-term processes (the campaign set the groundwork for future work). </li></ul>
    38. 43. Lessons Learned <ul><li>Evaluation </li></ul><ul><li>- needs to consider all campaign components and </li></ul><ul><li>contexts </li></ul><ul><li>- is not a simple one-off event at the end </li></ul><ul><li>of the project (evaluation should be addressed at </li></ul><ul><li>different stages of the program) </li></ul><ul><li>- planning requires an investigation of the most </li></ul><ul><li>appropriate methodology for the type of work being </li></ul><ul><li>evaluated (e.g. online survey versus a qualitative </li></ul><ul><li>evaluation using a focus group methodology; </li></ul><ul><li>different methodologies for each stage of program </li></ul><ul><li>development and implementation) . </li></ul>
    39. 44. <ul><li>Acknowledgements: </li></ul><ul><li>Aldo Spina – Evaluation Consultant </li></ul><ul><li>HIV/AIDS & Related Programs Unit, South </li></ul><ul><li>Eastern Sydney Illawarra Health (SESIH) </li></ul><ul><li>Thank you to the people who contributed to the development of the campaign materials. It is an act of generosity on their part to share their experiences with us. </li></ul>www.positivelife.org.au
    1. ¿Le ha llamado la atención una diapositiva en particular?

      Recortar diapositivas es una manera útil de recopilar información importante para consultarla más tarde.

    ×