Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Video Impact Assessment: Closing the Impact Gap

Last month I created an assessment for an international foundation on the impact of their video grants. The assessment (below) includes analysis and visualization of how 37 grantees used video in their advocacy work, and the effects of that use.

For this project I first analyzed grantee self-reports on how they were using video. I then developed a model comparing grantee activities and outcomes and finally visualized this analysis in the form of an easily skimmable “report card” for each grantee. This public version includes 10 representative reports from among the original 37.

  • Login to see the comments

  • Be the first to like this

Video Impact Assessment: Closing the Impact Gap

  1. 1. by Mary Joyce with John Kennedy for the Open Society Foundation | December 2015 (v1.1) | Part 1 Image: Flickr/Very Quiet VIDEO IMPACT ASSESSMENT Closing the Impact Gap Public Version
  2. 2. CONTENTS Executive Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . slide 3 Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .slide 4 Recommendations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .slide 7 Methods. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .slide 9 Grantee Report Cards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . slide 14 Appendices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .slide 54 This public version is anonymized to remove information identifying the nonprofit organizations being assessed and the foundation for which the assessment was prepared. The purpose of this anonymization is to protect the privacy of the entities involved and instead put the focus on the method of analysis and its presentation. Redacted content is covered with gray boxes. About the Public Version
  3. 3. * Assessment Part 1 includes a 51% non-random sample of 37 video-producing grantees.Image: Flickr/Very Quiet EXECUTIVE SUMMARY •This assessment of 2009-2015 grantees* revealed a gap between video content and video impact. •Despite strengths in storytelling, production, and distribution, grantees often fail to benefit from their videos. •Strategy is often rudimentary.While the vast majority of grantees understand the need for a goal and audience, most videos lack theories of change, precise audience targeting, and systematic measurement of impact. •To meet these challenges, the Media and Narratives Division should adopt a population benefit strategy that would focus funding priorities and methods of proposal evaluation and grantee support on videos that aim to achieve concrete benefits for marginalized populations.
  4. 4. Image: Flickr/Very Quiet FINDINGS
  5. 5. FINDINGS Impact is Little Understood & Poorly Measured Lack of Consensus on Impact •Impact claims were diverse, indicating little consensus on what constitutes video impact. •27% of grantees claimed that the video benefited the population on whose behalf their organization advocates, such as improved health services for Roma or kinder community treatment of HIV+ women. •14% claimed the video benefited their organization, such as helping them raise money or get invitations to speak at conferences. •32% of grantees claimed an effect on their audience, such as raising awareness or increasing understanding, though not a direct benefit to themselves or their population. •The remaining 27% made no claim of video effect or benefit, instead describing the scope of distribution or the quality of the video’s content. Population Benefit Organizational Benefit Audience Effect w/o Benefit No Effect or Benefit Claim n=37 32% 14% Impact Claims Shortage of Evidence •Many impact claims were unsupported by observable evidence, and none were measured systematically. •Of the 27 grantees who claimed an effect or benefit from their video, only 63% supported that claim with any kind of observable evidence, such as audience self-reports of opinion change or an audience member taking a desired action. •The other 37% of that group did not support their impact claims with any evidence or confused evidence of distribution, such as online view counts, with evidence of impact (effects and benefits). •0% of grantees engaged in systematic measurement of impact. Evidence was ad hoc and anecdotal. Note: See slide 55 for definitions of terms.
  6. 6. FINDINGS Strong Production/Distribution Hindered by Weak Strategy GreatVideo Content and Production •Grantees are telling compelling stories and producing high quality videos. •The overall quality of story and production is very high. •Selection of more compelling visuals and editing (to make them shorter and more conducive to online sharing) was a common suggestion for improvement. •Yet poor strategy often means that even excellent videos have little impact. Distribution Shines Despite Lack of Focus •Grantees are creative and energetic in their distribution, yet lack focus on specific audiences. •Grantees used a range of online and offline methods to distribute their videos. They used social media, webpages, and even temporary graffiti. They showed videos at workshops, conferences, trainings, public debates, and at meetings with decision-makers. •Their shortcoming was a lack of focus on audience. Of the 30 grantees with sufficient data to evaluate distribution, 70% did not connect their distribution methods to specific audiences they were targeting. •As a result, distribution was enthusiastic, but at times haphazard. The Strategic Deficit •Behind the impact deficit lies a strategy deficit. Grantees have mastered only the basics. •95% of grantees identified a goal and 86% identified one or more audiences, indicating widespread understanding of the most basic strategic principles of video advocacy. •Yet 32% are having problems identifying precise audiences, instead trying to reach the “general population” or “open public.” •Only 7 grantees (14%) were able to make an evidence-backed claim that their video created an audience effect that benefited either themselves or their population. •Outside of these 7, there was little evidence that grantees had in mind a video theory of change* that linked the audience effect of their video to benefits for the marginalized population. * Definition on slide 55, model on slide 56.
  7. 7. Image: Flickr/Very Quiet RECOMMENDATIONS
  8. 8. RECOMMENDATIONS Adopt a Population Benefit Strategy forVideo What is a Population Benefit Strategy forVideo? •This new Media and Narratives Division strategy would inform funding priorities, proposal evaluation criteria, and grantee support. 1.Funded videos would focus on achieving concrete improvements in the welfare of marginalized populations. 2.Funded videos would have a credible theory of change that describes the role the video will play in achieving that benefit. 3.Grantees would receive high- and low-touch support to gain the capacity to create video theories of change. Prioritize Advocacy Over Promotion •If population benefit is the goal, reconsider funding of promotional videos. •The videos in this assessment included both advocacy videos, which make an argument for the improvement in the welfare of marginalized populations, and promotional videos, which tout past successes of an organization. •Promotional videos help grantees become more sustainable by acting as a fundraising tool, but their connection to population benefit is indirect. •Advocacy videos provide a more direct effect on marginalized populations because that is their explicit goal. Focus on theVideo’s Theory of Change •Proposals should outline a series of causal steps that link video use to population benefit. •Grantees may need help seeing the big picture, as video impact and theory of change are still unclear to many. Provide a Suite of Support Services •Different grantees will need different levels of support to get to video theory of change. •For those with strong strategic thinking, more focused proposal feedback and visual aids (slides 12, 56) may be sufficient. •For those with weaker strategic thinking, technical assistance would be necessary. •Current strategic skills are summarized in the Grantee Report Cards, beginning on slide 14. Consider a Broader Assessment •This assessment is based on a 51% non-random sample of grantees who self-selected to report on their video impact. •Part 2 would analyze the impact of the other 49%, and give a more complete picture of grantee video impact.
  9. 9. IMPACT Image: Flickr/Tom Blackwell Findings Image: Flickr/Very Quiet METHODS Data & Model
  10. 10. DATA THE DATA Grantee Self-Reports Grantees wrote the following 3 self-report statements about each video they submitted. All 3 statements were analyzed for this assessment. 1) Description (Please provide a 50-100 word description of the video.) 2) Use Statement (How did your organization use this video in your advocacy work?) 3) Impact Statement (Did your video have a positive impact on your cause? If yes, what was that impact? If not, what prevented that impact?) TextsVideos Grantee-CreatedVideos A total of 72 videos were submitted to the Health Rights Online Film Festival and analyzed for this assessment. 2015 Health Rights Online Film Festival An online festival of grantee videos ( provided the perfect opportunity for an assessment of video advocacy. Of the 73 grantees invited to submit a video, 37 (51%) responded and are included in this assessment. They submitted not only videos, but also self-reports on both how they used those videos in advocacy and the impact they realized from that use. Both the videos submitted and the self-reports describing those videos were analyzed for this assessment. ANALYSTS Videos analyzed by John Kennedy Self-report texts analyzed by Mary Joyce
  11. 11. THE ANALYSIS Mary Joyce used grounded content analysis to analyze all 72 sets of video statements submitted by grantees. This qualitative method involved reading and annotating the 217 texts to identify patterns (image above). John Kennedy viewed the 72 videos associated with these texts to make his assessment.
  12. 12. Goal Desired effect on the audience and how this effect will lead to b e n e fi t s f o r t h e marginalized population. Audience Individuals whose change of mind or action would have a beneficial effect on the marginalized population. Story Narrative structures and characterization used to persuade the audience. THE MODEL Production Technical elements of the video, including sound, lighting, and editing. Distribution Means of ensuring the audience sees the video. Impact Extent to which the desired audience effect and population b e n e fi t s o c c u r r e d , a s demonstrated by observable evidence. For each grantee, analyses of both the video and texts are presented in an additive model of video advocacy impact. In this model, early choices on Goal identification and Audience specification lead to decisions about the Story a video will tell and the Production that will best express that story. The video is then Distributed according to the audience the grantee wishes to reach. Impact is evaluated by comparing the videos effect on the audience to the video’s initial goal. Analysis: Mary Mary Mary MaryJohnJohn cc cc Mary Joyce, 2015
  13. 13. CAVEATS Limitations of the Assessment • Selection Bias: The grantees self-selected to submit videos to the Health Rights Online Film Festival and to write the self-reports about those videos. As a result, the 37 grantees in this assessment represent a 51% non-random sample of the population of 2009-2015 video-producer grantees. • Positive Skew: This self-selection likely resulted in a positive skew on the data. Those who participated are likely more actively involved in video advocacy than grantees who did not participate. • Self-Reporting: The texts used in this assessment are written by the grantees themselves. One would expect a tendency to portray their own video work in a positive light. • Repurposed Data: The texts were produced as part of an application for the film festival’s Impact Award. Grantees were asked to speak about their video’s impact in general terms. They were not asked to respond directly to the elements in the model presented on the previous slide.
  14. 14. IMPACT Image: Flickr/Tom Blackwell Findings Image: Flickr/Very Quiet 1 GRANTEE REPORT CARDS
  15. 15. 1. African Palliative Care Association (APCA) 2. Association for Development and Social Inclusion (ADIS) 3. Association for Emancipation, Solidarity and Equality of Women of Republic of Madeconia (ESE) 4. Association for Supporting the Marginalized Workers (Star-Star) 5. Center for Interethnic Dialogue andTolerance Amalipe "AMALIPE" 6. Centre For Advocacy And Research / Center for Health and Social Justice (CHSJ) 7. Centro de Estudios para la Equidad y Gobernanza en los Sistemas de Salud (CEGSS) 8. Ceva de Spus 9. Children of Georgia 10.Citizen Association SOLEM 11.English Collective of Prostitutes 12.ENPUD,Alliance Ukraine 13.Eurasian Harm Reduction Network 14.Fundación Salud por Derecho 15.GeorgianYoung Lawyers Association (GYLA) 16.Gral Film 17.Health Education and Research Association (HERA) 18.Healthy Options Project Skopje (HOPS) 19.Hungarian Civil Liberties Union (HCLU) 20.International Committee on the Rights of Sex Workers in Europe (ICRSE) 21.Justice and Women CharitableTrust 22.Kenya Hospices and Palliative care Association (KEHPCA) 23.Kenya Legal and Ethical Issues Network on HIV and AIDS (KELIN) 24.Keystone Human Services Moldova 25.KHAM 26.New Zealand Prostitutes Collective (NZPC) 27.Pivot Legal Society 28.Rattapallax "DAVIDA" 29.Red Latinoamericana por el Acceso a Medicamentos 30.Roma SOS 31.Sex Worker Education and AdvocacyTaskforce (SWEAT) 32.Sexual and Health Rights of Marginalized Communities Coalition 33.Sonke Gender Justice 34.Transgender Equality Network Ireland (TENI) 35.Transgender Europe 36.Uganda National Health Users'/Consumers' Organization (UNHCO) 37.Ukrainian Community Advisory Board (UCAB) Patients of Ukraine THE GRANTEES Report cards are arranged in this section in the above order. Of the 37 grantees reviewed in the original version, a representative selection of ten are included here. They illustrate common challenges in making advocacy videos. 1. Filmmaking and Strategy are Difference Skills 2.The Importance of a Clear Goal 3.The Problem with “Raising Awareness” 4.The Importance of Active Distribution 5.The Problem withTargeting the “Public” 6. Impact Without Evidence 7. When Goal and Impact are Disconnected 8. Media Coverage ≠ Impact 9.VideoViews ≠ Impact 10. Simple Strategy Wins the Day
  16. 16. THE REPORT CARDS The assessment of each grantee’s video advocacy is presented in a report card, with above elements. Scores The grantee is scored on its mastery of each element of the model, as revealed through the videos and self-report texts included in the assessment. Quotes Grantee quotes from the self- reports are used whenever possible to explain the score. Recommendation Provides advice directly to the grantee, based on the model analysis. Videos Videos included in the assessment are listed, with links. Image A still from one of the videos included in this grantee’s assessment
  17. 17. Goal: I* “[P]eople with intellectual or psychosocial disabilities who a l s o h a v e p h y s i c a l disabilities... are confined... for two main reasons: there is... societal prejudice against them... and their physical e nv i r o n m e n t s d o n o t accommodate” them. This is great problem analysis. The next step is to figure out who has the power to address either reason and decide what action you want them to take. Then you’ll have your goal. Audience: C T h e “ p u b l i c ” i s identified, but this is an audience too v a g u e t o b e effectively targeted. Story: A Fantastic.The portraits are excellent. The i n t e r v i e w s w i t h spokespeople feel a little jarring - as they interrupt the flow of the portraits. Does Judy ever visit these institutions? I'm just looking for a visual way to link her to the very strong video. Distribution: A- Very active and diverse distribution methods - “Blog on OSF website, O S M H I F a c e b o o k , Yo u T u b e , V i m e o , International Film Festivals' screenings and public d i s c u s s i o n s , s h a r i n g between NGOs, TV broadcasting “ yet without an audience identified (or an action desired of that audience) it’s hard to know if these methods were well-selected. Impact: C- The impact described - “Public awareness, public discussions...” - seems to indicate your confusion as to what the film was meant to achieve and which e f f e c t s w o u l d b e e v i d e n c e o f t h a t achievement. Also, these are methods of distribution, not impacts (effects) of distribution. Production:A Such great filming and visual choices. Recommendation This is a fantastic video. Yet there’s a disconnect between filmmaking and strategy. The filmmaker is clearly talented at production, storytelling, and distribution. But without clarity on goal or audience it’s very hard to make an impact... and this is what happened to this film. The next time you work with an NGO, make sure they tell you exactly who they are trying to influence and what successful influence would look like to them. They may not be clear on the answers, so you may need to press them. The result, however, will be a more effective advocacy video. Videos in Assessment: New Day (2015): * Incomplete data GRAL FILMFilmmaking and Strategy are Different Skills1 This talented filmmaker was at a loss when it came to strategy.
  18. 18. CENTER FOR INTERETHNIC DIALOGUE AND TOLERANCE AMALIPE (AMALIPE) Goal: A- “We use the movie to make RHSP popular and to advocatte [sic.] for its sustainability” - The sustainability goal is clear, precise, and beneficial. Focus on that. Being “popular” is not precise and its benefit is also unclear. Audience: B+ This is a great example of how a clear goal leads to clear audiences, but unclear ones are a dead end. “[R]elevant national institutions” that could provide funding are a clear a u d i e n c e f o r t h e s u s t a i n a b i l i t y g o a l . Unsurprisingly, there is no goal for the vague goal of p o p u l a r i t y. A n d t h e “advocacy work at local level” - what is that about? Story: A- It looks more like a mini documentary about your work than an advocacy video as a tool. Production: A Really well filmed. Distribution: I* There’s little information on how you showed the video to the funding institution. Maybe this was because you thought a face-to-face meeting was boring or unworthy of mention. But distribution strategy does not need to be high-tech. It just needs to put your video together with its intended audience. And distribution for popularity? And the advocacy work? Need more info! Impact: C Impact assessment should link back to the goal using evidence to make an argument, but that didn’t happen here. There’s no mention of whether the video helped you get funding. You say that the video helped you popularize your program, but provide no evidence. You mention again advocacy at the local level, but this is a means of achieving impact, not the impact itself. Recommendation “Community Monitoring” is a very well filmed and edited video. However, it looks more like a mini documentary about your work than an advocacy tool. It’s hard to tell from the incomplete information, but it seems that you made this strategic analysis harder than it had to be. You needed funds. You created a video illustrating your past work to convince “relevant national institutions” to continue funding that work. That’s a completely acceptable video strategy! It just needs a means of measurement. Did you get funded? Did you receive more money than you had before you were able to use video as a promotional tool? Measurement of impact just means using observable evidence to evaluate the effect your video on your audience. If you keep your strategy simple it will be easier to both create video strategies and measure their impact. Videos in Assessment: Videos in Assessment: The White Swallows III (2015); Community Monitoring (2015) product/view/1/285.html * Incomplete data 2 The Importance of a Clear Goal A clear goal is the focus and foundation of video strategy.
  19. 19. Goal: B “To raise awareness toward disability issues.” - While a common goal of advocacy videos, raising awareness is not enough. What do you want to happen after people are aware? Sketch the full theory of change to keep yourself focused. Audience: I* “The video has a positive impact on a u d i e n c e - t h e audience is convinced” - There is an awareness that the audience is an impor tant par t of video strategy, but there is no actual identification of who the audience is. Story: A Very good storytelling. Continue to push your team - what are the best visuals and most important messages. (“All Boys” is my favorite video in the assessment - MJ) Distribution: A- The video was actively distributed on TV, social media, and through public events. You were also attentive to evidence of audience exposure, such as comments and likes. However, without a clear idea of who should be seeing the film it’s hard to tell whether the right distribution methods were used. Impact: B- I appreciate your honesty in saying you felt you couldn’t measure impact directly. This is because you did not identify a specific audience, so you didn’t know where to measure. If you select an audience, you will be able to measure impact because you will know where to seek evidence of effect. Remember that you cannot say a video had a “positive impact” if you have no evidence of audience effect. Production: A- Nice simple filming and titles. Grantee Recommendation Your film has a clear and excellent message and the filming is simple and well-done. Now it’s time to build on that by working on the other elements of your strategy so you can get a bigger impact next time. The next time you create a video, create your strategy first. Here are the basics of linking all the elements above into a clear strategy. The purpose of any advocacy video is to persuade an audience, either to take an action or to change their mind about something. The action/mental change they are persuaded to undertake is the goal. The argument the video makes to persuade them is the story. How you will make sure the audience sees the video is distribution. When you measure impact you are looking for evidence as to whether the audience took the action (easier to observe) or changed their minds (harder to observe). If you follow this strategic process you will get more value out of your future videos - and the ones you’ve already made. Good luck! Videos in Assessment: Institution or community? (2015); Returning Home (2015); I wish all boys from residential institutions could live in a family (2013) * Incomplete data KEYSTONE HUMAN SERVICES3 The Problem with “Raising Awareness” “Raising awareness” is too vague a goal for precise and effective strategy.
  20. 20. Goal: B “[T]o raise awareness around sex workers human rights and health issues, as well as the need for change in law” - You identify the direct effects of the video (raising awareness) and the ultimate impact you are aiming for (change in the law), but you don’t link them. Will raising awareness lead directly to changing the law? If not, what are the intermediate steps and how will you move through them? Audience: I* “Change in the law” i m p l i e s a g o v e r n m e n t a l audience, though none is specified. The individuals who would be targeted to sign the e-petition are also no specified. Story: A Very clear, well done. Impact: C- You write that you cannot measure impact yet, but since you do not even note the number of signatures on the e- petition, I suspect the number is lower than you would have wished, rather than being uncountable. You also “hope” the video “reach[es] people and bring[s] about change,” which is oddly passive given that it is the aim of your own campaign to achieve these ends. Don’t give up. You can make it happen! Recommendation Online advocacy is not a silver bullet. It just implies a new set of tools. Making an (admittedly high quality) digital video does not imply virality. Placing that video online does not guarantee engagement in an online tactic like e-petition signing. In fact, because there is so much content online vying for our attention, active promotion to clearly specified audiences is critically important. Although I sense that this effort might have been discouraging, you’ll get a better result if you engage in more intentional video strategy. Here are the basics: The purpose of any advocacy video is to persuade an audience to take action (or change their mind). The action they are persuaded to do is the goal. The argument the video makes to persuade them to take that action is the story. How you will make sure the audience sees the video is distribution. And when you measure impact you are looking for evidence as to whether the audience took the action (or changed their mind, though that is harder to measure). Good luck! Videos in Assessment: Decriminalise Sex Work Now!(2015): * Incomplete data SEX WORKER EDUCATION AND ADVOCACY TASKFORCE (SWEAT) Distribution: B The video was posted on its own page on your website, but this really isn’t enough. There is no m e n t i o n o f a c t i v e outreach (for example, via listservs and social media) to drive traffic to the page. You also do not specify the audiences you were trying to reach with each method. Production: A G o o d s i m p l e animation. 4 The Importance of Active Distribution Simply posting a video online isn’t enough to engage an audience.
  21. 21. CEVA DE SPUS Goal: A- “[T]o change the mentality of the public towards people with intellectual disabilities... t o p r o v e [ t h e i r ] abilities" - Ambitious, but I’d like to see the video linked to a more s p e c i fi c a n d measurable action from the audience. Perhaps t h e r e a c t i o n yo u w a n t e d f r o m t h e U n i v e r s i t y administrators you mention later? Audience: B “[T]he open public” - Trying to target the public is a common mistake. It’s too broad t o b e t a r g e t e d effectively. “[D]ecision makers... civil society or ganizations and donors” - These are a lot clearer. Story: A- Wonderful videos - so clear why we should all have equal rights. Think about more visuals. I know these were created as simple portraits - but I also know that Zoli is very skilled at capturing small moments which show the personality of a person. Production: A Good filming, Zoli! Distribution: A "[U]ploaded on social media and website... different events... in press releases.... projections at different events, round tables, conferences." Diver se and active distribution. Would benefit from a bit more l i n k a g e b e t w e e n distribution method and audience . Which audiences were you hoping to reach with e a c h m e t h o d o f distribution? Impact: A “The west University started an accessibility project, they make efforts to fulfill all of our demands... For sure this video had a small contribution to the declaration of Ministry of Labor regarding starting the deinstitutionalisation process” Though I wish there were some evidence behind the claim about the MoL, the p o l i c y c h a n g e a t t h e University is a great example about how focused pressure on specific institutions can be a starting place for broader social change. Recommendation We are both huge fans of this video series. You make it so obvious why no one should be institutionalized permanently. You’ve used video to realize concrete advances towards broad social change. Next time you make a video, be as specific as you can about who you are trying to persuade and what you want to persuade them to do or think. This will prevent you from trying to target overly broad audiences like “the open public.” Videos in Assessment: There is life outside the walls (2013); Andrei (2012); Raluca (2011) 5 The Problem withTargeting the “Public” This grantee made the common mistake of selecting an audience too broad to target effectively.
  22. 22. Goal: A+ A very clear goal: to help NGO’s in other countries learn from N Z P C ’ s b e s t practices as they m a k e s i m i l a r decriminalization effor ts. Clearly connects the effect of the video on the audience (learning) to the desired impact ( i m p l e m e n t i n g decriminalization internationally). Audience: A+ “ [ S ] e x w o r k e r o r g a n i s a t i o n s internationally” - A strategically logical audience given the goal. Story: B+ The message is clear as a report about the collective, but less of an advocacy message. Impact: B “We believe the video has had a positive impact as sex workers in other countries are using the information within it to build support for decriminalisation” - This could be true, but there’s no evidence provided to support your claim. (How do you know it is being used?) This is the only place where you video strategy falls short. The rest is excellent. Production: B+ Clear sound and visuals. Recommendation You are a great example of the strategic use of a promotional video. It is very rare for a video promoting an organization’s past work to have a clear and logical goal, audience, and distribution method, but you’ve done it. Congratulations! The place for you to focus now is on measuring your impact. You could easily get a sense of the impact of your video by following up with the people who ordered the DVD to see how they used it or by following up with anyone who contacted your through the video website. Videos in Assessment: Decriminalisation of Sex Work in New Zealand (2015): NEW ZEALAND PROSTITUTES COLLECTIVE (NZPC) Distribution: A+ Your distribution methods makes a lot of sense given your audience and goal. The video is available via DVD (perhaps for NGOs w i t h o u t i n t e r n e t connection) as well as on a purpose-built website. I also like that the video was actively distributed to other NGOs, rather than simply letting it sit on a website. 6 Impact Without Evidence This grantee’s strategy was really strong until the impact assessment stage, when they failed to measure the effect of their work.
  23. 23. ENGLISH COLLECTIVE OF PROSTITUTES Goal: B “1. to publicize our legal victories... 2. To build the m o v e m e n t f o r decriminalisation.“ - The value of the first goal is unclear. What future benefit will result from publicizing these past victories? The second goal is a bit vague. What exactly does “build the movement” mean here? How will the video accomplish this? Audience: C+ ”[M]any individuals and organisations” - Who exactly? Why is it important that they h e a r a b o u t yo u r victory? What role do t h e y p l ay i n t h e movement you are building? Story: B It's a very good video - but we need sex worker voices. Without their voices we can't be 100% sure this is what they are asking for. Distribution: B “[S]howing in December 2014” - Who was in the a u d i e n c e t h a t yo u wanted to persuade? “sent the film to our network” - To what end? What actions were you hoping for as a result? Impact: B- “We...have been invited t o s p e a k a t fi l m showings on a number of occasions.” - This beneficial outcome seems more by chance than by design. There’s a disconnect between goal and impact. Production: A Good filming - both interior and exterior (protest) shots are captured clearly. Recommendation I know you are awesome activists and I love both the spirit and the method of your activism, so I am going to be a little tough on you (as in touch love) regarding your video strategy. I get the sense that there wasn’t really a clear vision of what benefit you were hoping to get out of this video. As a result, the audiences targeted, the distribution method used to reach those audiences, and the impacts recorded seem rather haphazard. Bottom line - advocacy video is about persuading and audience to take an action (often mediated by a change in state of mind) and impact is about measuring the extent to which that action was taken. If you are specific about the action you want from the beginning, when you set your goal, the rest of the strategic process will be a lot easier. Videos in Assessment: SohoTrot (2015): 7 When Goal and Impact are Disconnected A big goal (movement-building) leads to minimal benefits (speaking invites) without clear causal steps connecting the two.
  24. 24. EURASIAN NETWORK OF PEOPLE WHO USE DRUGS (ENPUD),ALLIANCE UKRAINE Goal: B “ F o r d r a w i n g attention to the problem of clients of OST program in Donbass“ - Focus more on the action y o u w a n t t h e audience to take. Whose attention? To what end? Audience: I* “ [ D ] r a w i n g a t t e n t i o n ,” b u t whose attention? In t h e a b s e n c e o f audience analysis, realizing impact is very difficult. Story: A You’ve done a really good job eliciting stories from the protagonists. Their discussions of their families and emotional lives humanize them. I would have liked to see someone who is a recovering addict give a tour of the city he knew as an addict. Distribution: B+ “A lot of famous mass- media, such as BBC... key events of EEC region” - Though getting coverage from a media partner with a wide audience is impressive, this is only meaningful if the BBC’s audience is the audience you want to reach to achieve your goal. Also, remember that reaching an audience is evidence of distr ibution, not evidence of impact. Impact: B It’s possible that the media exposure resulted in actions helpful to the OST clients, but there’s n o e v i d e n c e . Remember that views are not impact, action is. Next time, think about who has the power to help and then target them specifically. Production: A- The filming is of good quality, but the visual choice (the candle, for example) could relate to the subject matter more directly. Recommendation Your passion about this topic really comes through and the suffering of the film participants is clear. For a crisis as challenging as this one, filmmakers need to connect to other partners who are capable of carrying out more direct (in-person) advocacy with audiences capable of providing practical assistance to those in need. If filmmakers are not currently connected to these actors, OSF could likely assist in making those connections. Massive humanitarian crises cannot be addresses by filmmakers working alone. They need help. Videos in Assessment: Slow Death in Ukraine (2015); Live or Die in Donbas? (2015);The First CrimeanVictims(2015);The Right to be Dependent - Irina (2011) * Incomplete data 8 Media Coverage ≠ Impact Media coverage is a mechanism of distribution, not a measure of impact. OST = opioid substitutions therapy
  25. 25. Goal: A- “ [ D ] e m a n d s f o r decriminalisation” - This is a good campaign goal (or even or ganizational mission), but I’d like to see a goal specific to the video. What role should t h e v i d e o p l a y i n convincing politicians to make this change? Do t h e y n e e d s p e c i fi c information? A more general emotional appeal to empathy to lay the groundwork for later policy arguments? Audience: B “[O]ur contacts... other networks and organisations” - I’d like to see more clarity about why e a c h a u d i e n c e needs to see the video. Story: B+ The fact that sex workers were willing to be filmed without anonymization is very rare. Yet the power of their self-advocacy is muted by the lack of a narrative arc. Distribution: B+ Online methods seems to be the primary means of distribution, including an online training and social media. Again, there doesn’t seem to be much clarity on who needs to see the video, thus also a lack of clarity on how to reach that audience. Impact: B- "I don't know. The video was seen 3,500 times (July 2015) so we assume it has allowed many people to make themselves familiar with our message." - I appreciate your h o n e s t y a b o u t y o u r uncertainty in measuring impact. I also like that you are looking for obser vable evidence This is the right direction. In the future, note that views are a measure of distribution, not impact. What did you want people to do after they received your message? Production: A- A nice use of natural light. Good looking interviews - but think about visuals that can also tell a story. Recommendation Your video contains truly excellent self-advocacy from a range of sex workers across the European region. Next time you create a video, create your strategy first. The purpose of any advocacy video is to persuade, so a video strategy needs to identify who needs to be persuaded (audience), what they need to be persuaded to do (goal), the argument the video will make to persuade them (story), how you will make sure the audience sees the video (distribution), and how you will know whether the audience persuasion took place (impact measurement). In particular, try to narrow down your goal to a specific change in state of mind or action needed from a specific audience, which will further your larger goal of decriminalization. This focus will help you develop a distribution plan linked more directly to specific audiences and you will have a clearer idea of whose reaction to the film you need to measure. Remember that it’s easier to measure action than changes in state of mind, so see if you can link the change you want from your audience into an observable action. Videos in Assessment: Sex workers are the solution not the problem - Sex work, HIV and human rights (2014): INTERNATIONAL COMMITTEE ONTHE RIGHTS OF SEX WORKERS IN EUROPE (ICRSE)9 VideoViews ≠ Impact Though they are easy to measure, video views are evidence of distribution, not of impact.
  26. 26. Goal: A+ “[T]o raise surrpport [sic.]” among “families [of] women living with Hiv/Aids” - A modest goal but one that is clearly tied to the video. “[T]o raise community awareness to counter stigma” - Clearly links the effect of the video (raising awareness) to b e n e fi c i a l i m p a c t s (reducing stigma, raising suppor t). Linking awareness raising to a broader purpose is so rare - great job! Audience: A+ “ [ C ] o m m u n i t y members... families [of] women living with Hiv/ Aids”” - The audiences are specific and are clearly tied to the goal. Excellent! Story: B+ Keep trying to find people who are willing to have their faces shown on camera. I know this is a sensitive subject, but there will be some champions who want to tell their story. Distribution: A+ “The video was viewed by over 1000 community m e m b e r s i n t h e i r homesteads where trained community volunteer s facilitated discussions on issues raised by women in the video.” - An excellent e x a m p l e o f l i n k i n g distribution method to audience in order to address a specific goal. Also excellent use of quantitative data as evidence. Despite the fact that the video is digital, distribution is all offline. This is exactly what what needed to reach your rural audience. Impact: A "The women involved in making the video and in facilitating the discussions reported qualitative changes with families showing increasing support to women living with Hiv/Aids.” - Using qualitative data (oral reports), this impact links back directly to the initial goal of the video. Again, it is rare to see this type of clear linkage. It would be nice to see some kind of impact assessment of the awareness- raising/stigma-reducing goal, and there is some confusion between distribution (views) and impact (action), but this is really good work. Production: A- Good visuals. Think about shortening the video - a shorter video will often be more powerful. Recommendation Congratulations! Your video has the strongest strategy of all the grantees in this assessment! Your goals, audience, distribution, and impact are all clearly linked. The use of evidence (views, oral reports) to support your impact claims is also excellent. You should be teaching other NGOs how to create video strategies! Videos in Assessment: TheTriple Burden (2015): JUSTICE AND WOMEN CHARITABLETRUST10 Simple Strategy Wins the Day This grantee’s simple strategy with clear causal linkages led to a beneficial outcome clearly connected to their use of video.
  27. 27. IMPACT Image: Flickr/Tom Blackwell Findings Image: Flickr/Very Quiet APPENDICES
  28. 28. • AdvocacyVideo: Recording of moving images designed to benefit a population or individual whose interests are marginalized or under-represented. • Audience: Individuals whose change of mind or action would have a beneficial effect. • Benefit: Concrete improvement in quality of life. • Distribution: Means of ensuring the audience sees the video. • Goal: Desired outcome. Video goals should reference both the direct effect on the audience and how this effect will lead to benefits for the grantee’s population of interest. • Impact: Extent to which the desired audience effect and population benefits occurred, as demonstrated by observable evidence. • OST: Opioid substitution therapy, a form of harm reduction in which users of opiates, such as heroin, are given substitution drugs, such as methadone, to manage their addictions. • Production:Technical elements of the video, including sound, lighting, and editing. • PromotionalVideo: Recording of moving images touting past successes, often for the purposes of gaining funds for future work. • Story: Narrative structures and characterization used to persuade the audience to undertake an action or change of mind. • Strategy: A plan for achieving a goal. • Theory of Change: A series of causal steps that explains why it is logical that the actions in the strategy will achieve the goal. While the strategy describes the actions of the advocate, the theory of change takes into account the broader context in which the advocate is operating. GLOSSARY Video AdvocacyTerminology
  29. 29. VIDEOTHEORY OF CHANGE Basic Model Desired Realized Marginalized Population Audience cc Mary Joyce, 2015 Goals & Planning Impact and Evaluation Campaign Goal Desired benefit for marginalized population 1 Video Goal Desired effect on audience 2 34 Video Impact Realized effect on audience Campaign Impact Realized benefit for marginalized population Start goal-setting here. End evaluation here. Campaign Video Compare to evaluate Compare to evaluate Audience action ID audience that can help population cc
  30. 30. Desired Realized Marginalized Population Audience cc Mary Joyce, 2015 Campaign Goal: Desired benefit for marginalized population 1 2 34 “Stop Sending TB Patients to Prison” by KELIN Video Goal: Desired effect on audience Video Impact: Realized effect on audience Campaign Impact: Realized benefit for marginalized population VIDEOTHEORY OF CHANGE Example: KELIN cc
  31. 31. IMPACT Image: Flickr/Tom Blackwell Findings Image: Flickr/Very Quiet 1 THANK YOU! Mary Joyce | Video Strategy Analysis, Findings and Recommendations, InformationVisualization and Models John Kennedy |Video Content and Production Analysis |