This document outlines the research plan for evaluating an OER Degree Initiative. It discusses conducting quasi-experimental studies on the impact of OER degrees on student outcomes at 10-12 partner colleges. It also involves collecting cost data through surveys, interviews and financial templates to analyze the cost impacts on students and institutions. The evaluation will examine academic and economic impacts through quantitative outcomes data and qualitative implementation research to provide formative feedback throughout the initiative.
Good morning! I hope you all received my email last week about being invited to join the RP breakout group this morning and that you all that as good news. The goal of this session is to provide a quick orientation to the research design and to make sure all of you feel comfortable with what is being asked of research partners.
Our goal is to provide an overview of what we want to learn, how we are approaching the research and evaluation, and delve a little into roles, responsibilities and timelines. We’ll also ask our partners at rpkGROUP to give a quick introduction to the cost analysis. If we don’t address all your questions in this session, don’t worry! We will be available via a sign up sheet during other sessions here in SF, and if there is demand we will host a webinar for research partners later in the summer. We’ll also be in direct communications with your team.
First, we’d like to start with some introductions… Ask one member from each team to stand up, identify your college, where you’re located, and the degrees you plan to offer.
We have an ambitious research agenda for this project. There are lots of strong hypotheses about the ways in which OER degrees will benefit students. Many of these were echoed in your proposals. For example… These are supported by varying degrees, from some empirical evidence to anecdotal. This initiative provides a fantastic opportunity to test these claims.
So these are our research primary research questions:
Gets at whether impacts on institutional culture come to fruition.
note that we’ll get to economic impacts in a minute)
To address these RQs we have four main components: 1. QE studies with research partners to estimate impacts. Will estimate for each college and conduct meta-analysis. 2. Research on academic impacts will be combined with cost analysis to measure cost effectiveness – ie are OER degrees a good way of allocating resources to improve student success? 3rd, we plan lots of data collection around implementation, such as surveys, interviews, and site visits. Also incorporate inputs from partners with on-the-ground presence, especially Lumen. Finally, we will aggregate pass/completion info to provide summative report on the impact of the initiative. Would be good to get similar data for comparable non-OER sections - won’t enable us to compare success to BAU, but may provide directional evidence.
In case all this isn’t totally clear yet, we’ve tried presenting these activities in a venn diagram. You can see that all 34 colleges will provide enrollment… For system grantees, we need to work out to what extent this can be handled through system lead. RPs participate in QEDs. Important point is that the implementation data collection cuts across both RPs and other colleges, as we want to learn about implementation in those that have less research capacity. (not that this is the case for all non-RPs, but it is not a random sample)
Let’s drill down for a second on RPs – why so important? As Karen said yesterday, this is the first systematic effort to get beyond individual course redesigns. The quasi-experimental studies we plan to conduct with RPs differ in an important way from existing research on OER effectiveness. Thus, we need to compose comparison groups of similar students, not courses.
Now, you may be thinking, “but wait, we also want to know about course level outcomes.” We support this line of inquiry as well! But we have to set priorities, so we are treating this as an optional add-on to quasi experimental studies
Contamination – note Karen’s comment that around 1/3 of grantees already have OER degrees