More Related Content Similar to Before the Surveys, Spreadsheets, and numbers: Laying a Solid Assessment Foundation (20) More from Shane Young (18) Before the Surveys, Spreadsheets, and numbers: Laying a Solid Assessment Foundation1. BEFORETHE SURVEYS, SPREADSHEETS, AND
NUMBERS
LAYINGA SOLIDASSESSMENT FOUNDATION
ShaneYoung
GLACUHO Annual Conference
October 22nd 2019
© ShaneYoung at www.sensiblyshane.com
2. BeforeWe Begin
■ Want a copy of this presentation?
– www.sensiblyshane.com > Presentations > Before the surveys […]
■ Want a copy of the documents referenced in this presentation?
– Email me at syoung@sensiblyshane.com
■ Looking for resources or want to share resources with the region?
– Check out our new GLACUHOToolkit (https://www.glacuho.org/page/Toolkit)
– Or Click on Resources >Toolkit
© ShaneYoung at www.sensiblyshane.com
4. Pictured above: What people think assessment feels like.
And admittedly how it sometimes is.
© ShaneYoung at www.sensiblyshane.com
5. ATale ofTwo Paradigms
■ Ewell (2009) points to two assessment paradigms
– Improvement
– Accountability
© ShaneYoung at www.sensiblyshane.com
6. What is assessment?
■ Banta and Palomba (1999) indicated that “assessment is the systematic collection,
review, and use of information about educational programs undertaken for the
purpose of improving student learning and development.
■ Banta and Palomba (2015), but more specifically Banta’s definition, says that
“assessment is the process of providing credible evidence of resources,
implementation actions, and outcomes undertaken for the purpose of improving the
effectiveness of instruction, programs, and services in higher education.
■ Angelo (as cited in Suskie 2009) defines assessment as “the ongoing process of:
– Establishing clear, measurable expected outcomes of student learning
– Ensuring that students have sufficient opportunities to achieve those outcomes
– Systematically gathering, analyzing, and interpreting evidence to determine how
well student learning matches our expectations
– Using the resulting information to understand and improve student learning
© ShaneYoung at www.sensiblyshane.com
7. What is assessment, but in red and
underlined!
■ Banta and Palomba (1999) indicated that “assessment is the systematic collection,
review, and use of information about educational programs undertaken for the
purpose of improving student learning and development.
■ Banta and Palomba (2015), but more specifically Banta’s definition, says that
“assessment is the process of providing credible evidence of resources,
implementation actions, and outcomes undertaken for the purpose of improving the
effectiveness of instruction, programs, and services in higher education.
■ Angelo (as cited in Suskie 2009) defines assessment as “the ongoing process of:
– Establishing clear, measurable expected outcomes of student learning
– Ensuring that students have sufficient opportunities to achieve those outcomes
– Systematically gathering, analyzing, and interpreting evidence to determine how
well student learning matches our expectations
– Using the resulting information to understand and improve student learning
© ShaneYoung at www.sensiblyshane.com
8. Evaluation
■ Scheerens, Glas, &Thomas (2003) define evaluation, generally, as “systematic
information gathering and making some kind of judgment on the basis of this
information” (pg. 3)
■ Suskie (2009) picks three definitions of evaluation
1. Judgement: Did we do the thing we wanted to do?What can we do better?What
did we do well?
2. Matching: Did intent match our impact?
3. Worth: Is our program doing well enough to continue?
© ShaneYoung at www.sensiblyshane.com
9. Evaluation continued
■ This term comes up often in assessment work and some people use the term
assessment and evaluation interchangeably.
– We’re going to assess whether or not you learned the thing.
– Our evaluation cycle has completed and now we can make some change
■ Theorists and researchers define and refine these terms regularly
■ My advice?
© ShaneYoung at www.sensiblyshane.com
10. What is the point?Why does assessment
matter?
■ Accreditation
– Accrediting bodies want to know more and more about the full breadth of the student
experience, including what and how they learn outside of the classroom
■ Students are changing
– Millennials are the entry and mid-level workforce
– Gen Z are your “traditional” college students (Hughey, 2019)
■ Budgets are under lock and key
– Why does your program deserve this funding more than X?
■ Improvement
– Even the best of the best needs to solicit feedback and make small adjustments
© ShaneYoung at www.sensiblyshane.com
11. Why does assessment matter?
■ Residence Life and Housing is trying to sell an experience, a lifestyle
– One that is better than other universities
– One that is better than off campus
■ What’s more compelling?
© ShaneYoung at www.sensiblyshane.com
13. -Made with Canva
“Student PNG image with transparent background”
by pngimg.com is licensed under - Creative
Commons 4.0 BY-NC
© ShaneYoung at www.sensiblyshane.com
14. ASSESSMENT IS ULTIMATELY ABOUT CREATING
A ROBUSTAND EVOLVING STORY OFA
DEPARTMENT, DIVISION, OR OFFICE
© ShaneYoung at www.sensiblyshane.com
15. I get it… Now when do I do the surveys?
■ There are plenty of things to do before getting to conducting surveys
The Assessment Cycle
© ShaneYoung at www.sensiblyshane.com
16. NACADA
The Assessment Cycle. [Image]. (n.d.) Retrieved from https://nacada.ksu.edu/Events/Assessment-Institute/The-Assessment-Cycle.aspx
© ShaneYoung at www.sensiblyshane.com
19. Shane’sAssessment Approach
■ Identify Purpose
■ Select Personnel
■ Set Program Goals or Outcomes
■ Develop and Implement Assessment Strategies
■ Review Findings
■ Create Action Plan
© ShaneYoung at www.sensiblyshane.com
20. PURPOSE
Adams, S. (2018, April). ‘Bad Data’ [Cartoon]. Dilbert. Retrieved
from https://dilbert.com/strip/2018-04-03
© ShaneYoung at www.sensiblyshane.com
21. Identify Purpose
■ This step is one of the most important steps
■ The purpose of assessment is to create a process in which an organization can improve
– In order to improve an organization it’s important to know what the purpose of
the organization is.
■ The purpose of assessment is not just for you, it’s for the entire department
© ShaneYoung at www.sensiblyshane.com
22. Where do organizations define purpose?
■ What is the mission of the organization?
■ What is the vision of the organization?
■ What does the organization do?
– How is that action reflected in the mission?
– How is that action reflected in the vision?
■ What does leadership believe is the purpose of the organization?
© ShaneYoung at www.sensiblyshane.com
23. Select Personnel
■ Assessment is not a one person job?
– Why?
■ One person = one subjective view
■ One person collecting, processing, and presenting data = madness
■ Why is this important to me? –Other employees
© ShaneYoung at www.sensiblyshane.com
24. (You) Select Personnel
■ Name three positions or people that should be
on an assessment committee
– Who did you pick and why?
© ShaneYoung at www.sensiblyshane.com
25. Selecting Personnel
■ One of the most important factors when creating an assessment team is to be at least
as representative as the campus population.
■ If the team building the data collection tools or reviewing the data does not represent
your campus population the assumptions made and the conclusions drawn will be
incomplete.
■ If your department is unable to create a committee that it representative of your
campus population… then there needs to be some time set aside to examine why that
is.
■ Based on the last activity and the people/positions you selected – would your
committee be representative of your campus population?
© ShaneYoung at www.sensiblyshane.com
26. Selecting Personnel
■ Other considerations
– New professionals
■ This is a great opportunity to reprogram them to understanding assessment is more
than crunching data
– Student Staff
■ “They wouldn’t be interested in this.” How do you know?
– Members from other departments
■ Is there a faculty member who specializes in educational assessment?
■ Not everyone has to be a “full member” they can be advisors to good assessment
practices
© ShaneYoung at www.sensiblyshane.com
27. Set Program Goals
or Outcomes
“Learning outcomes seem pretty
specific for a presentation on
laying a foundation…”
-You all, right now, probably
© ShaneYoung at www.sensiblyshane.com
28. Outcomes
■ According to Henning (2009) there are three main types of outcomes:
– Learning Outcomes:What will students learn?
■ Students will be able to articulate foundational management and leadership skills,
styles, and/or theories
– Operational Outcomes:What does our organization do?
■ Office of Student Engagement will plan and market weekly health and wellness
programming (Yoga, Zumba, etc.…)
– Program Outcomes: By simply existing what impact do we have
■ Increase in applications to and creation of student leadership roles
© ShaneYoung at www.sensiblyshane.com
29. Outcome
Alignment
■ Henning in (Allen, Elkins,
Henning, Bayless, & Gordon,
2013) posits that there should
be alignment from the top
(institution mission) to the
bottom (department goals)
■ I believe that this is
important… but is just a
framework for even better
assessment in the future
© ShaneYoung at www.sensiblyshane.com
30. Develop and ImplementAssessment
Strategies
■ This is step #4, but is often step #1 for a lot of organizations.
■ It is possible to start here and work backwards but that can lead to more frustration
■ Don’t reinvent the wheel (Suskie, 2011)
– Sometimes you already have awesome data collection tools.
Finck, N. (2010, April 9). The Commoditization & Fragmentation of the IA Community. [SlideShare] Retrieved
from https://www.slideshare.net/nickf/the-commoditization-and-fragmentation-of-the-ia-community© ShaneYoung at www.sensiblyshane.com
31. SURVEYS ARE COOL… BUTTHERE IS OTHER STUFFTOO!
© ShaneYoung at www.sensiblyshane.com
37. Don’t be scared
■ The first time is the most intimidating and the most difficult
■ When selecting personnel, did you select persons who were familiar with Excel or
other tools used for sifting through data?
■ If you followed the recommendations in the previous section – then go line by line and
fill with the data that you have.
– Operational Outcomes might be a good place to start
■ Reviewing findings should not be something that is done once a year
– This should be done when it makes sense
■ After a big survey push, for example
This Photo by Unknown
Author is licensed under CC
BY-SA
© ShaneYoung at www.sensiblyshane.com
38. CreateAction Plan
■ Did you meet the objectives that you created? Why?Why not?
– Were the objectives difficult to measure?
– Were the data collection methods not as effective as they needed to be
■ An annual review to prepare for the next academic year is a good idea, but discussion on
action items for next time should not be pushed off until the annual meeting
■ Do not reinvent the wheel
– Small adjustments over time are manageable and lead to less confusion with staff.
■ What your organization learned and that evidence should be transparent and
understandable to those within the organization at multiple levels, but also outside parties
(Bresciana, 2011)
© ShaneYoung at www.sensiblyshane.com
39. Barriers to Good Assessment
■ There is a reason that presentations on assessment are always sought
■ There are three main barriers to good assessment that I want to highlight
© ShaneYoung at www.sensiblyshane.com
40. You
■ Every single person in their
room can be a barrier to
good assessment practice
■ How might you be a barrier
to good assessment?
This Photo by Unknown Author is licensed under CC BY
© ShaneYoung at www.sensiblyshane.com
41. Other Staff
■ Thoughts other staff may have:
– “I had no part in this… why do I have to do this?”
– “I have no idea what this is… So I’ll just do it my way.” (Suskie, 2011)
– “Why do you need to know how my effective my office’s work is?”
– “Why are we doing this?”
■ Getting buy in from your staff is important in determining purpose, but is equally
important during each step
■ Assessment is a part of everyone’s role - but those leading assessment need to make
sure that others know that
© ShaneYoung at www.sensiblyshane.com
42. LEADERSHIP
Adams, S. (2011, August 17). ‘Untitled’ [Cartoon]. Dilbert.
Retrieved from https://dilbert.com/strip/2011-08-17
© ShaneYoung at www.sensiblyshane.com
43. Leadership
■ Leaders have their own
– Goals
– Criteria
– Outcomes
■ Sometimes the goals, criteria, and outcomes of a leader do not align with those of the
personnel chosen to complete assessment
■ Leadership may not be receptive to receiving negative feedback
© ShaneYoung at www.sensiblyshane.com
44. My Story About Assessment
■ Identify Purpose
■ Select Personnel
■ Set Program Goals or Outcomes
■ Develop and Implement Assessment Strategies
■ Review Findings
■ Create Action Plan
© ShaneYoung at www.sensiblyshane.com
45. Works Cited
■ Allen, K. R., Elkins, B., Henning, G. W., Bayless, L. A., &Gordon, T. W. (2013). Accreditation and the role of the student affairs professional. Washington, DC: ACPA-College
Student Educators International. Retrieved from http://www.myacpa.org/sites/default/files/accreditationmonograph.pdf
■ Banta, T. W., & Palomba, C. A. (2015). Assessment essentials: Planning, implementing, and improving assessment in higher education (2nd ed.). San Francisco, CA: Jossey-
Bass.
■ Bresciani, M. J. (2011, August). Making assessment meaningful: What new student affairs professionals and those new to assessment need to know (NILOA Assessment Brief:
Student Affairs). Urbana, IL: University for Illinois and Indiana University, National Institute for Learning Outcomes Assessment.
■ Ewell, P. T. (2009, November). Assessment, accountability, and improvement: Revisiting the tension (NILOA Occasional Paper No.1). Urbana, IL: University of Illinois and
Indiana University, National Institute for Learning Outcomes Assessment.
■ Finck, N. (2010, April 9). The Commoditization & Fragmentation of the IA Community. [SlideShare] Retrieved from https://www.slideshare.net/nickf/the-commoditization-
and-fragmentation-of-the-ia-community
■ Hughey, A. (2019, September 24). Generation Students: Challenges, Opportunities, and Best Practices [Webinar]. Studentaffairs.com. Retrieved from
https://www.studentaffairs.com/Webinars/generation-z-students-challenges-opportunities-and-best-practices/
■ Scheerens, J., Glas, C. A. W., & Thomas, S., Dr. (2003). Educational evaluation, assessment, and monitoring. [electronic resource] : a systemic approach. Swets & Zeitlinger.
Retrieved from http://search.ebscohost.com/login.aspx?direct=true&db=cat02173a&AN=akr.b5764476&site=eds-live
■ Suskie, L. A. (2009). Assessing student learning: A common sense guide. San Francisco, CA: Jossey-Bass.
■ Venaas, M. (2017, October 17). Storytelling in Assessment: Compelling Ways to Share Data. Retrieved from http://www.skyfactor.com/storytelling-in-assessment/
■ The Assessment Cycle. [Image]. (n.d.-a) Retrieved from https://nacada.ksu.edu/Events/Assessment-Institute/The-Assessment-Cycle.aspx
■ The Assessment Cycle. [Image]. (n.d.-b) Retrieved from https://www.northwestern.edu/searle/assessment-of-student-learning/assessment-process/index.html
■ The Assessment Cycle. [Image]. (n.d.-c) Retrieved from https://assessment.gwu.edu/assessment-cycle
■ Henning, G. W. (2009, March). Where are we going? What do we pack? Which road do we take? How will we know we arrived? Presentation at the ACPA annual convention,
Washington, DC.
© ShaneYoung at www.sensiblyshane.com
Editor's Notes Comedian Kiry Shabaz – If you know what I’m trying to say – don’t correct me. “Liberry vs. Library”
How you talk about how you’re going to assess or evaluate is a detail that should not take away from all the other work that goes into assessment. You’ve already built the box and now you’re trying to fit everything inside that one box in We tend to gravitate, create policies, procedures, and processes that are for persons who are like us and have similar beliefs Identify Purpose
Why does this committee exist? Do we do assessment? Do we manage assessment from the other committees? What are we primarily responsible for?
Select Personnel
I said “I like assessment” = chair of the committee
Staff who were 100% not interested
Set Program Goals or Outcomes
There were none. Or maybe there were some. At some point.
Trying to determine if we had any or needed to create some
Develop and Implement Assessment Strategies
While the former was going on – SSAO said “let’s do a survey!”
Guess who wrote the survey and had been using the exact same survey for years?
I want 90% response.
Review Findings
Create Action Plan