Assessment from Scratch: Using Logic Models to Guide Assessment Planning

Editor's Notes

1. Personal Introduction I’m from where Texas meets New Mexico & Mexico
2. Undergraduate degree from Boise State University
3. M.Ed. from Vanderbilt University
4. Fun Fact: My two postsecondary institutions match up with my initials.
5. We think we are teaching students many things… but what are they learning? How do we know?
6. By being here today and giving me an hour of your time – especially talking about assessment right after lunch! – I hope that you will all be able to…
7. This is a basic logic model. It lacks a lot of detail, but it is a framework demonstrates the three basic concepts of inputs, outputs, and outcomes. If we do this one thing – in this case the first year seminar – then we expect that first-year students will actually show up to meet with their peers each week. We expect that to result in persistence, maybe increased use of campus resources, etc. The logic model is used for one program or concept but it can be big or small. We could also say, if we have a fully functional office of academic advising, then all undergraduate students will meet with an advisor at least once per year, which should result in an increased 4-year graduation rate.
8. Now, let me introduce a slightly more complex logic model, but first I offer a disclaimer: This is only one version. The logic model is a tool for you to use as you see fit. You can add or subtract boxes, arrows, columns, etc. We are going to break it down one phase at a time. I’m not going to focus on the example, but on your handout there is one blank model, and one filled in. The model that is filled in is for a personal leadership development program, EVOLVE, at Vanderbilt. It is an 8-week program for a cohort of 30 transfer and sophomore students. At its core, this is similar to the If-Then model: we have inputs, outputs, and outcomes. We are just going into a little more depth about what each of these are.
9. First, Inputs. This is as obvious as it sounds. What does it take to make the program happen? Who needs to be involved? What will the real costs be? You should be specific here. Remember that the logic model is a summary, but does not have ALL of the information in one place. You do still need a real budget. For EVOLVE, the example that you have, the human resources included some campus partners in addition to our office staff. We spent a lot of time developing the program, had a few expenses, and definitely had to prioritize getting space at the right time during the year. This brings me to one thing that I want to point out: the logic model can be a fluid document. Some of our expenses were one-time because this program is new, so the cost in time and money will look different next time.
10. Outputs and outcomes may seem similar, but I do want to distinguish them. Outcomes are the learning outcomes or behavior modification that we expect to see from students. Outputs are the required steps to even reach the point where outcomes can be realized. The most direct way to think of this is that no matter what inputs we put into a program, how much time and money we spend, the learning outcomes are not realized if nobody SHOWS UP. First, let’s think about activities. In order for EVOLVEto happen, my office has to develop the actual programs we are going to facilitate. We sought feedback from campus partners about the program. We publicized it, and we held interviews for applicants. All of this is important to actually bring EVOVLE to reality, but designing curriculum and flyers is not the purpose or the goal. Participation is the critical role that others play. Even if we put in all of the inputs and all of these activities result, if NOBODY shows up, there is no more to our program. Be specific. Who will show up – are you targeting a specific population of students? Do you hope that faculty, staff, or community members will also “participate”? For EVOLVE, we targeted sophomore and transfer students. We needed those students to actually show up every week. We also invited staff partners to the students’ graduation event to hear their final presentations. The outputs are critical pieces of the program. Without them, the program does not exist. But they are not the PURPOSE of the program or the reason that it exists. We do not hold a student organization finance workshop so that we can make a handout or so that students sit in a room during lunch and eat cheap pizza. But making a handout and ordering cheap pizza are still outputs that belong in a logic model.
11. We hold the program so that students can learn how to manage money and take advantage of the institutional resources available to them. These are the outcomes with which we are probably all familiar. Learning outcomes belong in this section, but there can be much much more. The logic model tries to break these down and look at the time frame. I did not do that here because it is important to note that YOU define the time frame! For our EVOLVE program, we said that the short term outcomes would be the learning outcomes for each weekly session. The medium outcomes were those that related to the overall 8 week experience. Long term outcomes are anything beyond that. But again – your outcomes could look different. And, it’s okay if you don’t have any “long-term” outcomes!
12. When you put it all together, there will be some pieces that are more intricately woven together. That’s when you start drawing arrows. For example, if you have a presenter from the LGBTQI Life Center who is in the “input” category, that is probably more directly tied to the outcome of “awareness of resources for LGBTQI students”. You can draw an arrow however you like. You are only limited by your Excel skills and/or the number of marker colors in your office.
13. But I didn’t show you everything yet. Just to round things out, there are a few more pieces to consider. First, context: What is the context for this program? Another way of putting it might be, why does this program or opportunity need to exist? Typically there is some type of identified problem, need, or are for growth that you can identify and describe. This could be providing a service to transfer students. Context may also include important piece of information about your office or your students. For example, my office is only 1 year old. Part of the context of every program is that people on campus do not recognize our office name, so they might not think our programs are important/worthwhile. That also kind of leads into external factors: things you can’t control that could impact the success of your program. At Vanderbilt, nearly 50% of the students are fraternity/sorority members. We know that the new member experience is an external factor that will impact who participates in our programs during the spring after recruitment (not first years). The climate in your division of student affairs could also be an external factor. Assumptions ask us to look at what are we ASSUMING in creating this program? Typically we are assuming students will think this is worth their time, but there are some other pieces as well. It is worth pointing out that you should actually be thinking about these pieces BEFORE developing a program, if you are developing a new program, but I wanted to wait to talk about them until the end of the logic model description.
14. Now it’s time for me to acknowledge that I haven’t actually used the word assessment yet in this presentation. But I promise the whole thing has in fact been about assessment. The logic model is a tool for making sure that you can in some way demonstrate or measure whether you are doing what you say you are doing. If the logic model includes something, you should be able to measure or demonstrate it.
15. This does NOT mean that you have a survey question for every box! What it means is that you have a beautiful model that is supposed to represent, in summary, everything that is happening in this program from start to finish to long-term outcomes. You need to be able to demonstrate the reality of that. For the inputs, you can calculate how much money you spent. You can list the campus partners who helped. For outputs, you should have Facebook events that demonstrate the event was publicized, documents with the curriculum, and an attendance sheet or at least attendance count. Note that if you said you want faculty and students to participate, you should be able to demonstrate that both did… The outcomes are where our surveys tend to come in, at least for learning outcomes. But if one of your outcomes was that they give a presentation at the end, you don’t need to ask a question or get a copy to prove it happened. You saw it, and you wrote down the number of students that did it. Unless you want to do a qualitative word analysis, there’s no need to take it further. The other piece of the assessment conversation is that you DO NOT have a reason to ask survey questions or assess anything that is not in this model, because you have not identified it as important to the program. Of course we tend to always ask the open-ended “what is one thing we could do better?” question. That’s great, and it is a nice way to let students know that we care about their feedback. However, if you are asking 7 questions about how satisfied students were with different aspects of the program, and satisfaction is not an outcome, your assessment does not line up with how you are describing the program. To me, this is the ultimate value of the logic model. It is a metric for our assessment tools. When those are aligned with what we have outlined as important to the program, we have a whole picture that allows us to look at the program, figure out which pieces of the model are “working” as expected, and which ones aren’t.