1. University of Arizona
Scholarship Universe
Assessment Plan
A comprehensive guide to what SU hopes to achieve and how our work will
be evaluated, assessed, and reported.
Chris Elsner
4/9/2014
3. 2
Scholarship Universe Assessment Plan
Overview
When Scholarship Universe (SU) was initially created, the focus and emphasis was on creating a service
for UA Students – little attention was paid, at the time, to how we would assess and evaluate Scholarship
Universe. The need to report on the outcomes of SU since that time have resulted in the creation of a
series of ad-hoc and partial assessment tools. While these tools have provided useful data and allowed
us to assess SU to some degree, they don’t provide all of the information we need to accurately and
efficiently assess and report on the outcomes and impact that Scholarship Universe is having.
This document will provide a review of our assessment methods to date, including where they have fallen
short and lessons learned so far. We will outline the goals, outcomes, and impacts we wish to measure.
And finally we will lay out a comprehensive assessment plan on how we will measure the above, what
we’ll need to do it, and a tentative timeline for the implementation of the assessment plan.
Mission
The mission of Scholarship Universe is to provide a one-stop-shop for scholarships for UA students by
making it easier to find and apply for both external and internal scholarships, thus increasing the number
and value of scholarships that UA students receive each year.
Purpose
The purpose of our assessment efforts will be twofold: to improve site functionality, usability, and
engagement and to demonstrate the effectiveness of SU in terms of impact on students, departmental
users, and the University of Arizona in general.
Goals
3-5 high level outcomes we wish to achieve.
1. Improve student scholarship attainment by reducing the time it takes to find a scholarship, matching
students to scholarships they’re eligible for, and increasing the volume of vetted scholarships in our
database.
2. Continually improve the functionality of the Scholarship Universe site to increase ease of use and user
engagement.
3. Help students, as a whole, maintain or increase external funding levels.
4. Improve the efficiency of departmental scholarship processes, increase applicant quality and quantity,
and reduce the amount of undisbursed scholarship funding.
4. 3
Outcomes
What are the measurable actions we can assess to demonstrate progress on the aforementioned goals?
Outcomes Measurement Tool(s)
Maximize impact of time students spend on scholarships
(i.e. students should get the most out of their time searching
and applying for scholarships).
Survey of SU and Non-SU users (need benchmark
to compare)
Increase the avg. number of eligible scholarships for each
student.
Reporting in SU scholarships vs. students,
compared year on year
Increase the total number of scholarships in our database.
Reporting in SU on Live/Achieved/Pending
Scholarships – track year on year
Identify funding gaps based on scholarship eligibility
requirements vs. UA student attributes.
Attribute and eligibility reports within Scholarship
Universe on students and scholarships
Increase user engagement with SU.
Google Analytics: page views and time on site,
eventually include transactions and conversions
Improve user experience with SU.
User feedback surveys and eventually built in
survey/forms in SU
Increase overall amount of external funding students bring
in each year.
SSSP Baseline report
Maintain/improve overall external funding level brought in by
students.
UA Analytics – overall external funding amounts
and debt levels relative to increasing tuition
Increase efficiency of internal scholarship processes. Survey of SUDA users (pre/post?)
Increase applicant quantity and quality for internal
scholarships.
Survey of SUDA users, historical comparisons
Reduce the amount of undisbursed departmental
scholarship money.
UAF Reports on unspent scholarship funding each
year
Improve donor satisfaction with quality of applicants and
awardees.
Survey of donors via foundation and/or individual
departments
5. 4
Delivery of Outcomes
How will we / are we achieve(ing) those outcomes?
1. Maximize impact of time students spend on scholarships: We’ve already made searching for
scholarships much easier by matching students only to scholarships for which they’re eligible
based upon their profile question responses. We’ve also made it easier to apply for university
scholarships by allowing students, in many cases, to apply for multiple scholarships with one
application. To further reduce the time expenditure for students and maximize their return, we’ll
need to continue to improve the matching capabilities of SU, streamline the profile so it takes less
time to complete – or can be completed in sections, and tweak site functionality to improve user
experience.
2. Increase the avg. number of eligible scholarships for each student: By changing the way some
underlying functionality works with how we match students to scholarships, we’ll now be able to
measure the avg. number of scholarships a student is eligible for. To increase this, we’ll have to
continue searching for and adding external scholarships to SU in addition to brining on more UA
scholarships.
3. Increase the total number of scholarships in our database: Our student worker team will have to
continue to research and find external scholarships. We’ll also have to continue working with and
bring on new colleges/depts. to increase the number of internal scholarships which are available.
4. Identify funding gaps based on scholarship eligibility requirements vs. UA student attributes:
Recent changes to the structure of SU should enable us, for the first time, to analyze the eligibility
information of the student population as well as scholarships offered in SU. We will need to
develop and improve our reporting tools, but once those tools are in place we should be able to
more easily identify gaps between what scholarships are asking for and what students are
providing.
5. Increase user engagement with SU: With the addition of our Automatic Email System, which will
send automated updates to SU users, we’ll be able to provide students with notifications that
they have new scholarships that they’re eligible for, unanswered profile questions, or when
scholarship deadlines are approaching. This feature is expected to encourage regular usage of SU
since there currently aren’t a lot of reasons for students to log in frequently on their own.
6. Improve user experience with SU: First, we’ll continue to survey and solicit feedback from users
regarding the site and what’s working and what’s not. Secondly, we can analyze Google Analytics
data to identify when and where students leave the site to find bottlenecks or areas which
discourage use.
7. Increase overall amount of external funding students bring in each year: We provide an ever
expanding list of vetted external scholarships which students can be matched to. By doing so we
provide increased opportunities for students to find and apply for external scholarships. These
opportunities improve the chances that a student will be able to find a scholarship – the more
6. 5
scholarships students have available, the more they can apply to, and the greater the likelihood
will be that they’ll bring in external funding. We will continue to search for, vet, and add external
scholarships to our system to maximize the opportunities available to students.
8. Maintain/improve overall external funding level brought in by students: By continuing to
provide as many scholarship opportunities to students as possible, we can help to maintain overall
external funding brought in by students relative to increasing tuition costs.
9. Increase efficiency of internal scholarship process: We are currently achieving this outcome by
making SUDA available to colleges/departments at UA. SUDA provides a unified tool to collect
and review scholarship applications, make awarding decisions, and track scholarship awards over
time. There are numerous features in SUDA which increase the efficiency of the scholarship
process compared to what many colleges/departments are currently doing. We’ll continue to
improve the functionality of SUDA through feedback from our current SUDA users and the
development of new features.
10. Increase applicant quantity and quality for internal scholarships: By providing students with
single place to search for internal and external scholarships, and by providing students with the
ability to apply for internal scholarships, we can increase the number of applicants for internal
scholarships. The idea is that SU will become the go-to place for UA students to find scholarships.
Further, by matching students only to scholarships for which they’re eligible we reduce the
number of ineligible applications that colleges/departments must sort through thus improving
the quality of the pool of applicants they do receive.
11. Reduce the amount of undisbursed departmental scholarship money: In providing numerous
tools for the management of the scholarship process in one place, SUDA reduces the overall
amount of work required to administer such a process. This means that administrators in the
various UA colleges/departments can more successfully implement their scholarship processes.
Furthermore, by increasing the number of applicants we can help ensure that a
college/department will always have a qualified student to award scholarships too.
12. Improve donor satisfaction with quality of applicants and awardees: Working in conjunction
with the UA Foundation and various development officers within partner colleges/departments,
we’ll develop and implement a survey specifically for donors to gauge their overall satisfaction
with scholarship applicant pools and awardees. Ideally this survey will be conducted
longitudinally and include results from donors associated with departments both pre and post
adoption of SU.
7. 6
Measurement
How will we measure outcomes?
We’ll measure our outcomes through a variety of means including user surveys, data collected from
UAccess, data generated internally by the SU site, Google Analytics data, and data collected via social
media.
Direct
What are the things we can measure directly?
Much of the way students interact with and use the Scholarship Universe site can be measured directly
through internally generated data and Google Analytics data. We can also directly measure the total
amount of scholarship opportunities available through SU, the total value of external funding brought in
by students, and the value of external funding brought in by Scholarship Universe users. Finally we can
directly measure the amount of unspent scholarship funding that colleges/departments have left over
after each scholarship cycle.
Indirect
What are the outcomes that will require indirect measurement?
From some of the direct measurements above, we can also indirectly measure things like overall level of
external funding relative to tuition costs. By talking with colleges/departments and looking at previous
applicant pool data (when available) we can also measure the quantity and quality of the applicant pool
in SU.
Measurement Tools
What are the resources we currently have? Where are they located? What are their limitations?
Below are the primary measurement tools we have available to us (see appendix for more information):
Google Analytics (see appendix): this tool provides information relating to site usage. Currently,
it provides us with general information in the form of: number of site visitors, time on site, page
views, exit rates, source traffic, user devices, and location of users. In the future, we hope to
optimize Google Analytics to also provide information on usage patterns, flow of traffic, and to
capture the impact of email/social media campaigns. Google Analytics is an online tool – all data
is stored on and accessed from Google’s servers (raw data can be exported/downloaded). It’s
limited in that we cannot assess why people do what they do on the site or what they do after
they leave our site (for instance to apply for an external scholarship).
8. 7
Internally Generated SU Data: quite a bit of data is generated as a consequence of regular site
usage. This includes user lists, profile information, metrics for individual users, scholarship data,
user generated feedback, and user interactions with tools on the site. This data is stored on our
servers in SASG. A current limitation of SU generated data, is a lack of access to
reporting/analytics tools. Right now reporting tools are limited – our only real option is to pull
raw data from the site then attempt to analyze it via a 3rd
party application. However, in the
future it would be desirable to have some minimal built-in reporting/analytics tools to make this
information more accessible, easier to navigate, and easier to analyze.
User Surveys: we have access to and frequently use the Campus Labs survey toolset as our
primary means of survey collection. We’ve conducted numerous surveys of our core users in
addition to some peripheral user groups which are relevant the scholarship process. This is an
online resources made available to us through Student Affairs and data is primarily stored on
remote servers, though it must be downloaded in order to perform any sort of substantial
analysis. The primary limitation of surveys is participation. We’ve had some success in improving
response rate through monetary incentives, but this is cost prohibitive. Furthermore, we must be
very wary to use surveys sparingly to avoid survey fatigue among our respondents.
UAccess: this system provides us with financial aid information on every student at the UA. This
information provides big picture information about the flow of funding brought in by students
and helps us understand the impact of SU. All of the UAccess data are stored on UA servers. The
primary limitation of the data from UAccess is the lack of specificity relating to how external
awards are categorized in the system. In order to keep the system simple, the vast majority of
external scholarship awards are classified as ‘Donor Select’ which makes it attribution to a specific
scholarship impossible. However, by adding a step to the check processing process – we have
started linking external awards to Scholarship Universe so we can at least attribute it to SU, if not
to the exact scholarship in SU.
Social Media Analytics (see appendix): right now, our social media presences is somewhat
limited. It has not been a top priority and it’s not been necessary to achieve desired outcomes.
However, as time goes on it may be one of the most cost effective ways to reach students and
promote Scholarship Universe. Currently, we have Facebook and twitter accounts, however the
twitter account is somewhat underutilized. These resources are online, however data can be
exported to local machines for additional analysis. Our primary limitations are twofold: for one,
we somewhat lack the capacity and staff to fully take advantage of social media and two, the
information gained may not necessarily aid us in measuring out stated outcomes.
9. 8
Implementation
How will our measurement efforts be implemented?
Implementation Timeline
Some of our evaluation and assessment efforts are already underway, others are currently in
development, and a few still remain on the drawing boards to be implemented in the future. Below will
be a summary of what has already been established, followed by a rough outline of when additional
assessment tools will come into effect.
Existing Resources
UAccess – we’ve already established what data we need from UAccess (primarily the SSSP
Baseline Query in RPT). We recently added a comment option to UAccess to capture scholarships
that can be potentially associated with Scholarship Universe when UA receives checks. This
should enable us to provide a more accurate indirect measure of the financial impact of SU.
Google Analytics – google analytics has been set up for the current site and provides basic
superficial usage statistics which are adequate for now.
Survey Data – we have data from previous surveys which could be useful in future reporting as
well as providing a template for additional surveys in the future.
SU Generated Data – there is a sizable amount of data on SU’s usage in terms of internal metrics.
However, we have to find appropriate uses for it and figure out easier ways to get it out of the
system.
Future Resources
Create annual report template based upon this plan. This will potentially include analyzing
UAccess data with new comment code complete initial draft by May 2014.
Create, implement, and analyze a survey for SUDA users. To be completed by July 2014.
Update and refine Google Analytics site tracking abilities. To be completed concurrent with
website redesign process by January 2015.
Create a standardized student user survey to be used annually. Start in November 2014 and
complete initial survey round by March 2015.
Analyze AY 2014-15 UAccess data – first full year of data with new ID comment. Completed as
part of annual assessment – May 2015.
Develop and improve reporting and analytics tools in both SU Admin and SUDA. Tentative start
date of January 2015 and completion date of May 2015.
Develop randomized internal user feedback collection on SU. Tentative start date of January 2015
and completion date of May 2015.
10. 9
Annual Timeline
Below is a general guide to our yearly assessment cycle. It includes rough time frames for when major
reports are required as well as rough time lines for regular data collection activities and reporting.
January February March April May June
Annual
Student User
Survey Opens
Revise/review
assessment
plan.
Social Media
Review and
Planning
Google
Analytics
Review
Assess/prepare
student survey
Annual
Student
Survey Closes
Student
Survey
Analysis
Google
Analytics
Review
SUDA User
Survey Opens
Collection &
Analysis of
UAccess Data
Collection of
Internal SU
Data for AY
Annual AY
Reporting:
Scholarship
Value, Users,
Trends
Google
Analytics
Review
July August September October November December
Annual SUDA
Review
SUDA Survey
Closes
Prep for
SSF/FF
Reports
SSF/FF
Progress
Reports
Google
Analytics
Review
Social Media
review and
planning
Google
Analytics
Review
Collection of
Data for
SSF/FF Report
SSF/FF
Progress
Reports
11. 10
Reporting
How will we communicate our assessment results in terms of frequency, audience, and medium?
Scholarship Universe Annual Report
As mentioned above, we will start to compile and provide an annual Scholarship Universe report. This
report will coincide with end of the school year, allowing us to look at an entire academic year and
compare its results with previous years. This report will contain key findings and figures which may be
then used in additional documents/reports. This report will be geared primarily toward upper
administration officials. It will be created and distributed electronically with hard copies available upon
request. Additionally, elements of the report may be used for marketing purposes and end up in other
places.
Google Analytics Report(s)
Our Google Analytics report will be compiled every other month. It will compare usage statistics with the
previous periods as well as the same period in the previous year (when possible). The primary audience
for this report will be immediate supervisors. The report will be generated electronically and 1-2 page
executive summaries will be provided at the appropriate meetings.
SSF/FF Progress Reports
The SSF and FF progress reports will include key performance indicators in the appropriate context. The
audience for each of these reports is the corresponding student board. These updates are provided bi-
annually.
12. 11
Appendix
The appendix items have not yet been completed, but below you can see what we’re planning on
including for each section.
Measurement Tool Documentation
This section will contain detailed information on our various measurement tools. This will include the
following:
Step by step walkthrough for pulling data from each measurement tool,
Where and how the data is accessed from each measurement tool,
Who the contact person associated with each measurement tool is,
The process for verifying, organizing, and analyzing data for reports,
Any known limitations or issues with each measurement tool.
Google Analytics Assessment Plan
This section will contain a detailed plan for our Google Analytics reporting. It will include:
The specific things we’re trying to measure in Google Analytics,
Specific Key Performance Indicators (KPI’s) for the things we’re measuring,
How Google Analytics will be structured to capture information on the outlined KPI’s,
A plan for what our Google Analytics dashboard should include,
Instructions for reporting data and statistics from Google Analytics.
Social Media Assessment Plan
This section will match up with our forthcoming Social Media Plan and include information on:
Specific social media platforms currently utilized and potential areas of expansion,
How each social media platform interacts with and relates to SU and overall social media goals,
Instructions on how access analytical information from social media platforms in use,
Benchmarks from the past and goals for the future relating to user engagement with our social
media presence,
Ways to connect reported social media outcomes with overall SU outcomes.
