Call Girls In Shalimar Bagh ( Delhi) 9953330565 Escorts Service
Social Media Evaluation
1. Evaluating Social Media in
Extension Programming
National Association of Extension Program and Staff
Development Professionals
October 21, 2014
Sarah Baughman, Ph.D. & Brigitte Scott, Ph.D.
Military Families Learning Network
Virginia Tech
Sarah Baughman, Ph.D.
eXtension
2. Photo credit: Douglas Wrey on http://www.geek.com/wp-content/uploads/2012/02/social_media_donut.jpg
8. • Support F2F workshops with information on
Facebook
• Use a FB page to encourage discussion on
educational information presented F2F
• Provide FB incentives for person who increases the
most or tries something new
• Have participants each take a different day to
share one recipe they have tried or something
from their food log that is working (or maybe not
working)
• Invite family members to the FB page to encourage
participants
•Highlight a different family member every week and
how they have helped support healthier eating
18. Photo by LauraGilchrist4 - Creative Commons Attribution-NonCommercial-ShareAlike License https://www.flickr.com/photos/76060406@N07 Created with Haiku Deck
21. Basic Text Analysis: Inductive
Use data to discover concepts, themes,
or models
Evaluator as interpreter; highly involved
22. Basic Text Analysis: Inductive
Use data to discover concepts, themes, or
models
Evaluator as interpreter; highly involved
Emergent, “bottom up”
23. Basic Text Analysis: Inductive
Use data to discover concepts, themes, or
models
Evaluator as interpreter; highly involved
Emergent, “bottom up”
Qualitative outcome: key themes or categories
relevant to evaluation/research questions
33. Basic Text Analysis: Deductive
Data is analyzed according to prior
assumptions
Evaluator is “independent” from data
34. Basic Text Analysis: Deductive
Data is analyzed according to prior
assumptions
Evaluator is “independent” from data
A-priori; “top down”
35. Basic Text Analysis: Deductive
Data is analyzed according to prior
assumptions
Evaluator is “independent” from data
A-priori; “top down”
Quantitative outcome: metrics relevant to
evaluation/research objectives
36. Application: Deductive Analysis
Category comparison, comparison over
time
Analyzing webinar chat pods
Analyzing how a hashtag is leveraged in
Tweets
Facebook/LinkedIn audience
engagement
37. Basic Deductive Analysis: 4 Steps
1. Develop data categories.
2. Clearly define those categories.
3. Read through all raw data and apply
categories.
4. Count.
38. Chat Pod Engagement Metrics
21
0
17
10
5
0 5 10 15 20 25
Unique participant to participant exchanges
Participant questions
Resources shared by MFLN
Resources shared by participants
Unique chat pod participants
39. The fine print….
Only DCO viewers can participate in the chat pod; percentage of chat pod participants based on
total number of DCO viewers and total number of unique participants.
Resources shared by participants include shared links, authors, studies, books, etc.; demonstrates
high-level engagement because participants are contributing to the co-construction of knowledge
during webinar.
Resources shared by MFLN include links, peer-reviewed studies and books, etc., from both MFLN
and non-MFLN authors; demonstrates direct CA engagement with participants by further
supporting and contextualizing knowledge construction by situating webinar presentation within
the larger disciplinary area.
Participant questions are those listed in the chat pod; demonstrates intent to pursue two-way
engagement in webinar and therefore high-level engagement.
Unique participant to participant exchanges are those in which chat pod participants respond
directly to one another’s comments; demonstrates high-level engagement through realized
reactive (two-way) and interactive (dependent) discourse patterns.
Chat pod text related to webinar content is not captured as an engagement measure due to its
discursive category as declarative (one-way) communication. (It is noted, however, that
declarative text is still understood to indicate webinar engagement, and MFLN encourages and
values such participant engagement.)
Chat pod text related to technical issues and/or CEUs is not included in MFLN evaluation.
42. Storytelling
Identify narratives that connect to your
evaluation aims
Be strategic and leverage stories for
evaluation task at hand
43. Storytelling
Identify narratives that connect to your
evaluation aims
Be strategic and leverage stories for
evaluation task at hand
Contextualize your stories with other data to
show a larger picture
44. Storytelling
Identify narratives that connect to your
evaluation aims
Be strategic and leverage stories for
evaluation task at hand
Contextualize your stories with other data to
show a larger picture
Ethics, ethics, ethics
46. From the Master Gardeners…
“On a Celebrex commercial a guy is
shown bent over in some beets or chard
and he raises up with a beautiful eggplant!
The first time I laughed at it my wife
thought I was crazy.”
47. Application: Storytelling and
Evaluation
Use stories in your reports, and include an executive
summary of those stories
Incorporate compelling stories with facts and figures
Include stories with direct quotes in press releases, on
Web sites
Include stories and quotes in newsletters, brochures,
annual reports
Social media has become a “catch all” phrase for lots of different ways we communicate in the virtual world. The key to understanding and conducting social media measurement and evaluations is to understand that social media is a tool. There are lots of different ways we can use the tool and that’s what drives the evaluation process.
Here are a few popular social media sites that Extension educators may be using. An important consideration when talking about measuring or evaluating social media work is how social media is being used
So there are lots of different purposes for social media –
Perhaps the most obvious purpose is marketing but it can also be used as part of educational programs, as a “stand-alone” type of program or event or for evaluation and accountability.
Here is a nice example of the Extension Master Gardeners using FB as part of their educational programming – the original post was about the spread of africanized honey bees and this screen clip shoes part of the discussion happening about africanized honey bees.
This shows an annual report from instagram for a small regional zoo.
So, as is always the case with evaluation – measuring and evaluating social media is contextual and there is no one way to do it “right”
More typically, Extension educators are going to use social media as part of their educational programming. So, for example they may use Pinterest to post recipes for a nutrition education program or perhaps they are using instagram as part of a teen leadership program. In this case, they will need to be able to MEASURE their social media activity as part of the overall program evaluation.
Let’s walk through an example of what that might look like for a fictitious nutrition education class: The primary outcome for the program is to increase consumption of fruits and vegetables for the participants.
So here is a very basic theory of change – we’ll invest resources and conduct activities with the goal of increasing participants consumprtion of fruits and vegetables
The program activities are going to include providing educational material on nutrition and the benefits of eating healthy. Participants will also be asked to maintain a food log and family members will be enlisted to support participants.
Now we need to understand some more specifics of our program and target audience before we can align social media strategies with the programmatic goals. For the purpose of this example nutrition program I am going to arbitrarily say this is a 6 week nutrition program for low income women.
For this program I have chosen to use Facebook because of the target demographic, women and presumed ages btw 18-39 from underserved audiences. Additionally, FB has good mobile version that allows all of these things to happen and increasingly more women are using smartphones, especially Hispanic and African American women. Finally, these social media strategies are all designed to enhance the face to face workshops, not replace them. Facebook is simply another program tool, it’s not a stand alone program.
So, if a program participant does not have FB they will not miss out on the educational programming.
Additionally if you are in an urban area serving primarily younger Hispanic or African American women I would consider using Instagram as well. One way to do this would be to have participants post pictures of their bfast one week, lunches the next, snacks, dinner, etc. .
With a more suburban audience that includes young white women I would consider using pinterest.
So what kind of data will we use to help measure success or programmatic impact?
I’m going to talk about some strategies for doing basic qualitative evaluation of social media.
A few things to keep in mind:
--Context: I’ll be talking about a few qualitative methodologies, but I’m presenting them in a very basic way specifically for the social media context. If you are looking to do qualitative evaluations utilizing some of the more traditional methods, such as interviews, focus groups, open-ended surveys, participant observation, these methods need to become more involved. And I’ll address that toward the end of my presentation.
--Rapid feedback: I’ll be talking about strategies that are very basic not only because I’m operating under the assumption that you’ll be using them in small chunks of social media (i.e., text comments from a Facebook post), but also in such a way that you can get quick insights on your social media strategy and social media impact. We all know that qualitative work can often be very time consuming. And while the strategies I’ll talk about will take a little longer than running Facebook insights or exporting a report from Sprout Social, they will not represent a huge time investment. So I hope you give them a try.
--Caveat: While I’m talking about some quick and dirty strategies here, I do want to state that I do not want to be undermining the aims of qualitative research, which is generally to interpret the lived human experience. This is a loaded phrase. So while I’m giving you some tips here, I don’t mean for the tips to undermine qualitative research as a complex, transformative field of inquiry.
--Strategy: Don’t forget about the importance of a social media strategy. Using qualitative analysis in social media presumes that you have an established presence in a particular place. If you don’t have one, or need to develop one, you might not quite be ready for qualitative social media evaluation. Since we’re trying to capture experience, impact, reaction, and the like, you need to first create an environment on social media where your target audiences are coming to discuss, respond, engage. MFLN has been up and running for 4 years now, and we’ve used that time to develop a presence on social media. But we are still working on making our sites interactive. So we’re changing our sm strategy to begin really focusing on that interaction piece and to embark on some more qualitative evaluation work within our social media sites.
I’m going to start by talking about some basic text analysis, with two different approaches.
The first approach is inductive analysis. What do I mean by that?
The WHY behind a highly engaging FB post
The WHAT behind a successful Tweetchat or hashtag
The WHY/WHAT behind comments/conversations around a blog post or a LinkedIn discussion
C/p from FB: You get a horribly formatted doc, shown above with a filter to protect identity.
End up with three files: raw data, participant key, “cleaned” data file (obviously “participant key” needs to be dealt with ethically; for sm eval, I advocate deleting this file UNLESS part of your SM strategy is to qualitatively track participants over time with types of post they respond to and/or types of comments they make, if you would need/find this data useful, etc.
Raw data file: Clear all formatting, remove pictures, insert naming conventions established in participant key (I use R1, R2, R3, etc.). Be sure to pay attention to these so that you can tell when participants are responding to one another—may be useful in a deductive analysis later! I also leave the number of likes and the time stamps, which may be useful to you if you want to see the types of comments participants resonate with, and the time lapse of how the conversation unfolds.
Insert spacing between comments or whatever you need to keep organize/sane.
Add hard returns to represent replies to comments.
Clean data file: Good to check cleaned data against FB; sometimes the number of likes come after the “reply,” etc.
Reading through your data is the very first level of analysis. Get familiar with your data, read it several times, and gain an overall sense of what’s happening, content being discussed, things like tone, resonance, disagreement, etc. And now is also the time to start taking notes on what you are seeing and understanding. Do NOT skip the notetaking! I promise you it will come in handy as you move through the next steps. This is also an important step toward credibility/triangulation: a record of your thought process will be helpful as you discuss your findings, and because it provides evidence that you have not simply read the data, wrote your reaction, and called it “analysis.” This is a part of inductive analysis, which is step-wise and systematic.
Decide how you are going to track and organize your codes. I’ll show you one of my typical Excel strategies in the next slide, but you can also use the comment feature in Word, color coding in Word, and if you’re more of a visual/tacticle thinker, use sticky notes, crayons, etc. Whatever works for you.
So now we really start to get into it.
Here’s what my coding process looks like when I use Excel. I put all the text (comments) I’m analyzing into the Comments section. The participants are also in there. I like to color code chunks of data when it makes sense. So here, I code each comment and the replies it gets into one color. This is a nice visual way to easily see the types of comments that are compelling to folks. And don’t forget to put the text of your post in—it should be a part of your analysis as well.
So you’re going to take several runs at coding. I like to code in the morning when I’m drinking coffee, and then return at different points in the day. Depending on the amount of text you’re coding, you’re going to want to return to it over several days to se The idea is that you’re not coding just to get it over with. You’re coding to get maximum understanding. And because you’re doing emergent, bottom-up work here, you want to keep returning to your codes and the data to see if your perspectives change, if you get new insights, etc. This is related to the constant-comparative approach of data analysis, which was pioneered by Glaser and Straus (1967), is central to grounded theory. You’re not doing grounded theory work here in this basic inductive approach primarily because you are only working with one static data set. However, I’m a big fan of constant-comparative analytical methods because I think they do ensure a certain level of rigor and credibility—you’re not coding once, but you’re coding several times. Adjusting and taking notes as you go. This leaves a really nice audit trail. So in your spreadsheet, you might start with one code (or several) for one chunk of text, but then change it two days later. This is the spirit of constant-comparative analysis. So if you change a code, make a note in whatever way makes sense to you, and don’t delete it from your file. Keep it. I often just use the comment function in Excel for all of this, but do what makes sense for you.
Higher-level, initial codes: often initially related to evaluation aims for your programming; this makes sense. As the evaluator, you have your logic model on your mind, you ideally have a social media strategy with goals and benchmarks on your mind, so these will inform your codes. This is great. But at some point, things are going to get a little richer, and little deeper, and start reflecting the actual content and dynamics of the discussion.
Lower-level codes: after multiple readings of data, may end up being related to actual words or phrases in the text.
Not all text will be coded.
So you’re done coding, you’re sick of reading, you have lots of notes. The next step is to go back through your codes and check for overlap. This happens all the time when you’re doing open coding like this. So if you have several “technology” codes, take a look at them. The goal here is not necessarily to simplify, although a bit of simplification is a natural outcome of this step. This is another step in rigor: e.g., are there differences in your technology codes? If so, what are they? How does this change your coding?
Capture key aspects of data that have emerged (in your view) as most important/informative. Go back and read your notes! Make connections across codes. What themes are emerging? How do you know? This is where it all comes together. And again, while this considered the last step, it doesn’t mean necessarily that it will be quick.
I have something very important to say: Writing is a level of analysis!!!!!! Don’t forget this!
So you’ve gone through and read, coded, recoded, taken notes, created categories. Don’t go and put those categories into an Excel graph! WRITE about them! I am willing to bet that it is in this writing process that you gain your deepest insights. Take your time here. And don’t forget to put a little bit of yourself in your analysis: that is, use “I,” briefly explain your process, tell your story as the interpreter of the data.
Your narrative analysis should stand alone, but it’s also great practice to through in a data viz section with the “metrics” of the post: number of likes, shares, comments, etc. And, you can do a triple whammy and do a companion deductive analysis of the same post. I’ll talk about that process next.
Who is talking to who during webinars? Are they sharing resources? What kinds?
Is your hashtag used in conversations? To share resources?
Is your audience using your page to discuss content related to your programming?
In deductive analysis you’re going to start with your categories. Notice the huge difference here vs. starting with your data and coding it. You can develop your categories without even collecting data when you do deductive analysis. Your categories should be related to your programming goals and your social media strategy. So if your goal is to make Facebook it’s own little community of practice based around your programming, you’re going to define key indicators, or categories, that support it’s development toward that goal.
Then you need to clearly define those categories. Very clearly, such that nothing is left up for interpretation.
Then you apply to your data, and count. Let me show you an example from MFLN.
Again: Stay organized, and keep notes
Not all text may fall into categories.
You may choose to use inductive analysis for uncoded text.
You may have noticed the rising popularity of storytelling lately, particularly in the context of evaluation: it’s powerful. Storytelling is a foundational way human beings make sense of the world. Every day we share our own experiences and engage in the experiences of others in multiple mediums: in conversation; on the Internet; in books, newspapers, magazines, movies, and social media; on television; in pictures. Our stories are everywhere because they are powerful expressions of the lived human experience.
Storytelling has its roots in qualitative inquiry
Narrative inquiry/analysis: focus on understanding the lived experience
Storytelling can be contextualized in other modes of evaluation:
Surveys
Interviews
Case studies
Social media evaluation
I want to share some guiding principles for collecting and utilizing stories. You may develop your own more systematic approach as you stay attuned to narratives as they emerge from your target audience.
Tips for storytelling in evaluation:
--Positive, negative, or anywhere in between
---Not always success stories; may illustrate problems, or successes, or barriers to success, may even “dramatize” the needs/assets/resources of the communities you serve
--Combine your stories with other data in your reports. This will provide for a full range of perspectives and overall will strengthen the quality of your evaluation reporting. Combinations of data can also work to underscore the impact of your chosen story.
--Confidentiality is important, particularly over social media. You may wish to contact the person directly to ask if you can share her/his story in an evaluation report; some will argue that discourse via social knowledge is in the public domain. You need to make this decision for yourself, but certainly sensitive topics should be handled differently than stories shared over “casual” conversation.
We’ve been talking about some qualitative strategies to use in your social media evaluation. We’re apply basic concepts from qualitative methodologies and leveraging them in the context of social media. I do want to be very clear: if you are doing a qualitative research study, or are conducting interviews, focus groups, participant observations, etc., as part of your evaluations, there are definitely layers of complexity and depth that will need to be added in to your methods. I am NOT suggesting with this webinar that qualitative work across the board is simple, quick, and easy, or that it can or should be done lightly. However, I am of the position that context matters. And basic qualitative evaluation methods for social media can be simpler and quicker than many might assume. So my goal today has been to get you thinking about adopting some of these strategies. But before I wrap up I want to return to a few larger issues I mentioned at the start. These larger issues tare a part of all qualitative work, and still need to be considered with the strategies I’ve discussed today.
Reflexivity in your qualitative work simply means that you are thinking about yourself as researcher/evaluator, and how YOU may impact the analysis and findings. It’s thinking about who’s done the research, how, why, under what circumstances. I don’t want you to get too caught up in this, but just take a few steps to think critically about your relationship to your work, how you do it, and what you produce.
I talked earlier about the importance of taking notes during your analysis process. And I want to remind you again here: transparency brings validity to our work. Period. But to be truly transparent, you need to be systematic in your analysis, diligent in your notetaking, and reflexive along the way.
We’re building here, as reflexivity and transparency together, as I just said, enforce credibility.
And finally, ethics. Social media is often considered to be a part of public knowledge. Personally, I’m completely comfortable doing an inductive analysis of a FB post like we’ve done here, knowing I’ve completely anonymized the participants, will not keep the participant key, and always further anonymize or completely leave off any potentially identifying information that gets shared in the comments and replies. However. If you are going to start asking for stories, tracing the the experiences of particular individuals who participate in your programming to show change or impact over time, etc., get consent. Be professional, ask for participants, go through IRB. There is a line tin the sand for sure. So even through we’re still in the social media context, evaluation work still needs to be ethical, and it’s up to you to ensure you’re protecting the people you are serving.
